Uh oh!
There was an error while loading. Please reload this page.
Optimize scan with a lookup table to reduce branching - #59106
Closed
Daniel Rosenwasser (DanielRosenwasser) wants to merge 13 commits into
Closed
Optimize scan with a lookup table to reduce branching#59106Daniel Rosenwasser (DanielRosenwasser) wants to merge 13 commits into
scan with a lookup table to reduce branching#59106Daniel Rosenwasser (DanielRosenwasser) wants to merge 13 commits into
Conversation
MemberAuthor
It's probably faster to actually check the common case of spaces, tabs, and newlines, but the current draft always does the lookup. TypeScript Bot (@typescript-bot) perf test this |
Contributor
TypeScript Bot (typescript-bot)
commented
Jul 2, 2024
Contributor
Daniel Rosenwasser (@DanielRosenwasser) Here they are:tscComparison Report - baseline..pr
System info unknown Hosts
Scenarios
tsserverComparison Report - baseline..pr
System info unknown Hosts
Scenarios
startupComparison Report - baseline..pr
System info unknown Hosts
Scenarios
Developer Information: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
MemberAuthor
TypeScript Bot (@typescript-bot) perf test this |
Contributor
TypeScript Bot (typescript-bot)
commented
Jul 2, 2024
Contributor
Daniel Rosenwasser (@DanielRosenwasser) Here they are:tscComparison Report - baseline..pr
System info unknown Hosts
Scenarios
tsserverComparison Report - baseline..pr
System info unknown Hosts
Scenarios
startupComparison Report - baseline..pr
System info unknown Hosts
Scenarios
Developer Information: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
MemberAuthor
TypeScript Bot (@typescript-bot) perf test this |
Contributor
MemberAuthor
TypeScript Bot (@typescript-bot) perf test this |
Contributor
TypeScript Bot (typescript-bot)
commented
Jul 2, 2024
Contributor
Daniel Rosenwasser (@DanielRosenwasser) Here they are:tscComparison Report - baseline..pr
System info unknown Hosts
Scenarios
tsserverComparison Report - baseline..pr
System info unknown Hosts
Scenarios
startupComparison Report - baseline..pr
System info unknown Hosts
Scenarios
Developer Information: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
TypeScript Bot (typescript-bot)
commented
Jul 2, 2024
Contributor
Daniel Rosenwasser (@DanielRosenwasser) Here they are:tscComparison Report - baseline..pr
System info unknown Hosts
Scenarios
tsserverComparison Report - baseline..pr
System info unknown Hosts
Scenarios
startupComparison Report - baseline..pr
System info unknown Hosts
Scenarios
Developer Information: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
MemberAuthor
I'm actually not sure if that's much better or not, it could just be a better run. TypeScript Bot (@typescript-bot) pack this |
Contributor
Contributor
Hey Daniel Rosenwasser (@DanielRosenwasser), I've packed this into an installable tgz. You can install it for testing by referencing it in your and then running There is also a playground for this build and an npm module you can use via |
scan with a lookup table to reduce branchingMemberAuthor
TypeScript Bot (@typescript-bot) pack this |
Contributor
MemberAuthor
TypeScript Bot (@typescript-bot) test top400 |
Contributor
Contributor
Hey Daniel Rosenwasser (@DanielRosenwasser), I've packed this into an installable tgz. You can install it for testing by referencing it in your and then running There is also a playground for this build and an npm module you can use via |
TypeScript Bot (typescript-bot)
commented
Jul 2, 2024
Contributor
Daniel Rosenwasser (@DanielRosenwasser) Here they are:tscComparison Report - baseline..pr
System info unknown Hosts
Scenarios
tsserverComparison Report - baseline..pr
System info unknown Hosts
Scenarios
startupComparison Report - baseline..pr
System info unknown Hosts
Scenarios
Developer Information: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Daniel Rosenwasser (DanielRosenwasser)
marked this pull request as ready for review
July 2, 2024 20:30
TypeScript Bot (typescript-bot)
commented
Jul 2, 2024
Contributor
Daniel Rosenwasser (@DanielRosenwasser) Here are the results of running the top 400 repos with tsc comparing Everything looks good! |
MemberAuthor
Titian Cernicova-Dragomir (@dragomirtitian) maybe this balances out the parse-time hit from #58928. 😄 |
Daniel Rosenwasser (DanielRosenwasser)force-pushed
the
scan-experiments
branch
from
July 3, 2024 19:58
27392e7 to
33ae58dCompareMemberAuthor
TypeScript Bot (@typescript-bot) perf test this |
Contributor
TypeScript Bot (typescript-bot)
commented
Jul 3, 2024
Contributor
Daniel Rosenwasser (@DanielRosenwasser) Here they are:tscComparison Report - baseline..pr
System info unknown Hosts
Scenarios
tsserverComparison Report - baseline..pr
System info unknown Hosts
Scenarios
startupComparison Report - baseline..pr
System info unknown Hosts
Scenarios
Developer Information: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||
| const tokenCategoryLookup: TokenCategory[] = []; | ||
| const tokenCategoryLookupUncommon = new Map<CharacterCodes, TokenCategory>(); |
Contributor
There was a problem hiding this comment.
How does map lookup performance and memory usage compare to branching ifs with comparison operators?
| [CharacterCodes.replacementCharacter, TokenCategory.RecognizedMisc], | ||
| ] | ||
| ) { | ||
| if (key < tokenCategoryLookup.length) { |
Contributor
There was a problem hiding this comment.
Why do this in a branch? The uncommon lookups could just be passed to the Map constructor.
There was a problem hiding this comment.
Can you be more specific? In the first two commits, I split this out from a map to an array and map to avoid the map lookup.
Contributor
There was a problem hiding this comment.
I mean that, since we know ahead of time which characters belong in the Map, we can just construct the map with those character codes rather than looping and comparing. It's not that important, and if it makes more sense to have them visually grouped together, that's perfectly fine.
| } | ||
| if (!(tokenCategory & TokenCategory.RecognizedMisc)) Debug.fail(`Unhandled token category ${tokenCategory}`); | ||
| switch (ch) { |
Contributor
There was a problem hiding this comment.
I wonder if the switch here is optimized in any way or if it is doing case-by-case comparisons against each character code. If the latter, maybe it would be worthwhile to analyze what that most common character codes in JS are and change the order of the cases so that the more frequent cases are earlier in the list. I expect the perf improvement here actually comes from handling digit and identifier characters earlier than the switch, since those are the most frequent characters in a given file. If that's the case, some of the complexity and memory overhead could be reduced by just optimizing for those two cases.
There was a problem hiding this comment.
I think it is typically case-by-case comparisons. You're absolutely right that doing the checks earlier on for common classes; however, I think that checking for all the identifier starts isn't practical (or requires the same sort of fast-path/slow-path separation where we check for ranges), so that's why the token categorization is helpful here. By dismissing all the other types of categories, we can immediately launch into the fast path.
Contributor
There was a problem hiding this comment.
Assuming CharacterCodes is a const enum, the VM should be able to optimize the switch case quite heavily as a branch table, but that's a question for V8.
Contributor
There was a problem hiding this comment.
If the majority of identifier/keyword characters are ASCII characters, then it's fairly trivial to check for those major cases without an array or Map using a simple comparison:
if((ch|32)>=CharacterCodes.a&&(ch|32)<=CharacterCodes.z||ch===CharacterCodes._||ch===CharacterCodes.$){/* [A-Za-z_$] */}If the array/map lookups are faster than this for the identifier case, then by all means keep it as is. I only point this out as you explicitly avoid hitting the table for line terminators and spaces. If that was an important optimization, then it might make sense to do the same kind of optimization for the most frequent (and easy to optimize for) characters in a source file, e.g., ASCII letters.
Contributor
There was a problem hiding this comment.
Assuming
CharacterCodesis aconst enum, the VM should be able to optimize the switch case quite heavily as a branch table, but that's a question for V8.
We've generally found that not to be the case, which is why we've had to implement our own jump tables for functions like forEachChild. I'm not sure what heuristics V8 uses for switch optimization, if any.
There was a problem hiding this comment.
If the majority of identifier/keyword characters are ASCII characters, then it's fairly trivial to check for those major cases without an array or Map using a simple comparison:
Fair, and this is what Anders Hejlsberg (@ahejlsberg) also suggested. The same way whitespace is prioritized, ASCII range is also very likely and worth prioritizing as a quicker up-front case instead of hitting the lookup table. It would mean there's some code duplication but maybe it's fine.
There was a problem hiding this comment.
Assuming
CharacterCodesis aconst enum, the VM should be able to optimize the switch case quite heavily as a branch table, but that's a question for V8.
We've generally found that not to be the case, which is why we've had to implement our own jump tables for functions like
forEachChild. I'm not sure what heuristics V8 uses forswitchoptimization, if any.
Especially in the case where these character codes are so sparse - I am very doubtful that V8 would do such an optimization, and at best I'd guess we'd end up with something like a binary search.
| if (tokenCategory & TokenCategory.SimpleToken) { | ||
| pos++; | ||
| return token = tokenCategory & TokenCategory.SimpleTokenMask; |
Contributor
There was a problem hiding this comment.
A comment here would be good for future readers as it isn't obvious at first glance how a TokenCategory becomes a token.
Comment on lines
+2067
to
+2101
| if (ch === CharacterCodes._0) { | ||
| if (pos + 2 < end && (charCodeUnchecked(pos + 1) === CharacterCodes.X || charCodeUnchecked(pos + 1) === CharacterCodes.x)) { | ||
| pos += 2; | ||
| tokenValue = scanMinimumNumberOfHexDigits(1, /*canHaveSeparators*/ true); | ||
| if (!tokenValue) { | ||
| error(Diagnostics.Hexadecimal_digit_expected); | ||
| tokenValue = "0"; | ||
| } | ||
| else { | ||
| pos++; | ||
| tokenValue = "0x" + tokenValue; | ||
| tokenFlags |= TokenFlags.HexSpecifier; | ||
| return token = checkBigIntSuffix(); | ||
| } | ||
| else if (pos + 2 < end && (charCodeUnchecked(pos + 1) === CharacterCodes.B || charCodeUnchecked(pos + 1) === CharacterCodes.b)) { | ||
| pos += 2; | ||
| tokenValue = scanBinaryOrOctalDigits(/* base */ 2); | ||
| if (!tokenValue) { | ||
| error(Diagnostics.Binary_digit_expected); | ||
| tokenValue = "0"; | ||
| } | ||
| return token = SyntaxKind.NewLineTrivia; | ||
| } | ||
| case CharacterCodes.tab: | ||
| case CharacterCodes.verticalTab: | ||
| case CharacterCodes.formFeed: | ||
| case CharacterCodes.space: | ||
| case CharacterCodes.nonBreakingSpace: | ||
| case CharacterCodes.ogham: | ||
| case CharacterCodes.enQuad: | ||
| case CharacterCodes.emQuad: | ||
| case CharacterCodes.enSpace: | ||
| case CharacterCodes.emSpace: | ||
| case CharacterCodes.threePerEmSpace: | ||
| case CharacterCodes.fourPerEmSpace: | ||
| case CharacterCodes.sixPerEmSpace: | ||
| case CharacterCodes.figureSpace: | ||
| case CharacterCodes.punctuationSpace: | ||
| case CharacterCodes.thinSpace: | ||
| case CharacterCodes.hairSpace: | ||
| case CharacterCodes.zeroWidthSpace: | ||
| case CharacterCodes.narrowNoBreakSpace: | ||
| case CharacterCodes.mathematicalSpace: | ||
| case CharacterCodes.ideographicSpace: | ||
| case CharacterCodes.byteOrderMark: | ||
| if (skipTrivia) { | ||
| pos++; | ||
| continue; | ||
| tokenValue = "0b" + tokenValue; | ||
| tokenFlags |= TokenFlags.BinarySpecifier; | ||
| return token = checkBigIntSuffix(); | ||
| } | ||
| else { | ||
| while (pos < end && isWhiteSpaceSingleLine(charCodeUnchecked(pos))) { | ||
| pos++; | ||
| else if (pos + 2 < end && (charCodeUnchecked(pos + 1) === CharacterCodes.O || charCodeUnchecked(pos + 1) === CharacterCodes.o)) { | ||
| pos += 2; | ||
| tokenValue = scanBinaryOrOctalDigits(/* base */ 8); | ||
| if (!tokenValue) { | ||
| error(Diagnostics.Octal_digit_expected); | ||
| tokenValue = "0"; | ||
| } | ||
| return token = SyntaxKind.WhitespaceTrivia; | ||
| tokenValue = "0o" + tokenValue; | ||
| tokenFlags |= TokenFlags.OctalSpecifier; | ||
| return token = checkBigIntSuffix(); | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
Suggested change
| if(ch===CharacterCodes._0){ | |
| if(pos+2<end&&(charCodeUnchecked(pos+1)===CharacterCodes.X||charCodeUnchecked(pos+1)===CharacterCodes.x)){ | |
| pos+=2; | |
| tokenValue=scanMinimumNumberOfHexDigits(1,/*canHaveSeparators*/true); | |
| if(!tokenValue){ | |
| error(Diagnostics.Hexadecimal_digit_expected); | |
| tokenValue="0"; | |
| } | |
| else{ | |
| pos++; | |
| tokenValue="0x"+tokenValue; | |
| tokenFlags|=TokenFlags.HexSpecifier; | |
| returntoken=checkBigIntSuffix(); | |
| } | |
| elseif(pos+2<end&&(charCodeUnchecked(pos+1)===CharacterCodes.B||charCodeUnchecked(pos+1)===CharacterCodes.b)){ | |
| pos+=2; | |
| tokenValue=scanBinaryOrOctalDigits(/* base */2); | |
| if(!tokenValue){ | |
| error(Diagnostics.Binary_digit_expected); | |
| tokenValue="0"; | |
| } | |
| returntoken=SyntaxKind.NewLineTrivia; | |
| } | |
| caseCharacterCodes.tab: | |
| caseCharacterCodes.verticalTab: | |
| caseCharacterCodes.formFeed: | |
| caseCharacterCodes.space: | |
| caseCharacterCodes.nonBreakingSpace: | |
| caseCharacterCodes.ogham: | |
| caseCharacterCodes.enQuad: | |
| caseCharacterCodes.emQuad: | |
| caseCharacterCodes.enSpace: | |
| caseCharacterCodes.emSpace: | |
| caseCharacterCodes.threePerEmSpace: | |
| caseCharacterCodes.fourPerEmSpace: | |
| caseCharacterCodes.sixPerEmSpace: | |
| caseCharacterCodes.figureSpace: | |
| caseCharacterCodes.punctuationSpace: | |
| caseCharacterCodes.thinSpace: | |
| caseCharacterCodes.hairSpace: | |
| caseCharacterCodes.zeroWidthSpace: | |
| caseCharacterCodes.narrowNoBreakSpace: | |
| caseCharacterCodes.mathematicalSpace: | |
| caseCharacterCodes.ideographicSpace: | |
| caseCharacterCodes.byteOrderMark: | |
| if(skipTrivia){ | |
| pos++; | |
| continue; | |
| tokenValue="0b"+tokenValue; | |
| tokenFlags|=TokenFlags.BinarySpecifier; | |
| returntoken=checkBigIntSuffix(); | |
| } | |
| else{ | |
| while(pos<end&&isWhiteSpaceSingleLine(charCodeUnchecked(pos))){ | |
| pos++; | |
| elseif(pos+2<end&&(charCodeUnchecked(pos+1)===CharacterCodes.O||charCodeUnchecked(pos+1)===CharacterCodes.o)){ | |
| pos+=2; | |
| tokenValue=scanBinaryOrOctalDigits(/* base */8); | |
| if(!tokenValue){ | |
| error(Diagnostics.Octal_digit_expected); | |
| tokenValue="0"; | |
| } | |
| returntoken=SyntaxKind.WhitespaceTrivia; | |
| tokenValue="0o"+tokenValue; | |
| tokenFlags|=TokenFlags.OctalSpecifier; | |
| returntoken=checkBigIntSuffix(); | |
| } | |
| } | |
| if(ch===CharacterCodes._0&&pos+2<end){ | |
| switch(charCodeUnchecked(pos+1)){ | |
| caseCharacterCodes.X: | |
| caseCharacterCodes.x: | |
| pos+=2; | |
| tokenValue=scanMinimumNumberOfHexDigits(1,/*canHaveSeparators*/true); | |
| if(!tokenValue){ | |
| error(Diagnostics.Hexadecimal_digit_expected); | |
| tokenValue="0"; | |
| } | |
| tokenValue="0x"+tokenValue; | |
| tokenFlags|=TokenFlags.HexSpecifier; | |
| returntoken=checkBigIntSuffix(); | |
| caseCharacterCodes.B: | |
| caseCharacterCodes.b: | |
| pos+=2; | |
| tokenValue=scanBinaryOrOctalDigits(/* base */2); | |
| if(!tokenValue){ | |
| error(Diagnostics.Binary_digit_expected); | |
| tokenValue="0"; | |
| } | |
| tokenValue="0b"+tokenValue; | |
| tokenFlags|=TokenFlags.BinarySpecifier; | |
| returntoken=checkBigIntSuffix(); | |
| caseCharacterCodes.O: | |
| caseCharacterCodes.o: | |
| pos+=2; | |
| tokenValue=scanBinaryOrOctalDigits(/* base */8); | |
| if(!tokenValue){ | |
| error(Diagnostics.Octal_digit_expected); | |
| tokenValue="0"; | |
| } | |
| tokenValue="0o"+tokenValue; | |
| tokenFlags|=TokenFlags.OctalSpecifier; | |
| returntoken=checkBigIntSuffix(); | |
| } |
There was a problem hiding this comment.
It's hard to parse out what the suggestion is - can you explain it?
Contributor
There was a problem hiding this comment.
Ah, that's fair, I don't know why GitHub insists on showing the old code in the diff rather than diffing against your PR version of the code.
Suggestion is to promote the pos + 2 < end check to the same if that checks for CharacterCodes._0, since all branches beneath it use that condition, then replace the nested if statements with a single switch (charCodeUnchecked(pos + 1)) with grouped case statements for each of the radix markers.
| } | ||
| if (!(tokenCategory & TokenCategory.RecognizedMisc)) Debug.fail(`Unhandled token category ${tokenCategory}`); | ||
| switch (ch) { |
Contributor
There was a problem hiding this comment.
Assuming CharacterCodes is a const enum, the VM should be able to optimize the switch case quite heavily as a branch table, but that's a question for V8.
MemberAuthor
#59244 tries to just check for identifiers up-front, and moves checks for |
TypeScript Bot (typescript-bot)
commented
Mar 24, 2026
Contributor
With 6.0 out as the final release vehicle for this codebase, we're closing all PRs that don't fit the merge criteria for post-6.0 patches. If you think this was a mistake and this PR fits the post-6.0 patch criteria, please post to the 6.0 iteration issue with details (specifically, which PR and which patch criteria it satisfies). Next steps for PRs:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change introduces two lookup tables to the lexical scanner/tokenizer. One is a simple array-based lookup for entities in the ASCII range, and the other is a
Mapfor wider codepoints.These tables assign categories to codepoints to indicate how they should be handled. For example, all line breaks, single-line whitespace, and single-character tokens can be handled generically in their own respective fashion. Doing so allows us to perform fewer individual checks to jump into or skip over a given branch.
One additional optimization is the way in which simple single-character tokens are handled is that the token itself is encoded in the lower 8 bits the category value. This allows us to avoid any branching for such cases.
This all comes at the expense of some memory overhead, along with some overhead from fetching from the table. To avoid hitting the table too often, spaces, tabs, and carriage returns, and line feeds are currently handled separately.