Uh oh!
There was an error while loading. Please reload this page.
Tokenizer's Interfaces Cleanup - #7001
Conversation
tarekgh
commented
Feb 15, 2024
@michaelgsharp I appreciate it if you could review the changes. I have removed a couple of APIs you introduced earlier and provided a workaround for their usage. Thank you! |
tarekgh
commented
Feb 15, 2024
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@## main #7001 +/- ##
==========================================
- Coverage 68.81% 68.80% -0.02%
==========================================
Files 1258 1258 Lines 250477 250652 +175 Branches 25576 25602 +26 ==========================================
+ Hits 172377 172468 +91 - Misses 71473 71553 +80 - Partials 6627 6631 +4
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Uh oh!
There was an error while loading. Please reload this page.
| internal List<Token> TokenizeWithCache(string sequence) | ||
| { | ||
| List<Token> tokens = new(word.SymbolsCount); | ||
| Word word; | ||
| if (Cache is not null) | ||
| { | ||
| if (Cache.TryGet(sequence, out word)) | ||
| { | ||
| return WordToTokens(ref word); | ||
| } | ||
| foreach (Token token in word.GetIterator(VocabReverse)) | ||
| word = MergeWord(sequence); | ||
| Cache.Set(sequence, word); | ||
| } | ||
| else | ||
| { | ||
| tokens.Add(token); | ||
| word = MergeWord(sequence); | ||
| } | ||
| return tokens; | ||
| return WordToTokens(ref word); |
There was a problem hiding this comment.
Can this whole method just be:
List<Token>result=new();TokenizeToIdsWithCache(sequence,result);returnresult;?
There was a problem hiding this comment.
No, TokenizeToIdsWithCache(sequence, result); fill only the Ids and removing the overhead for filling the whole tokens. Note the difference in the implementation between TokenizeToIdsWithCache and TokenizeWithCache. The first is calling WordToIds while the second is calling WordToTokens
| /// <param name="sequence">The sequence to split.</param> | ||
| /// <param name="isSpecialToken">Indicate if the token is a special token.</param> | ||
| /// <param name="accumulatedIds">The list of accumulated tokenized Ids.</param> | ||
| public override void TokenizeToIds(string sequence, bool isSpecialToken, IList<int> accumulatedIds) => TokenizeToIds(sequence, accumulatedIds); |
There was a problem hiding this comment.
When do we use the word Tokenize vs the word Encode?
There was a problem hiding this comment.
We use the Encode name at the Tokenizer class level to process the entire input text. At the level of the tokenizer's models, we use the Tokenize name, which operates on smaller, pre-tokenized text units.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
stephentoub
left a comment
There was a problem hiding this comment.
There are a few things that can be tweaked further, but that can be done in a follow-up.
This update encompasses the following:
Tokenizer.GetEncodedIdsCountAPI, essential for supporting crucial scenarios and implemented it in all supported tokenizers.EncodeToIdsandGetEncodedIdsCounthas been customized for other tokenizer models likeBpeandEnglishRoberta. This adaptation aims to enhance the performance of these APIs specifically when invoked from those respective tokenizers.