Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model - #7425

Merged
tarekgh merged 3 commits into
dotnet:mainfrom
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Mar 19, 2025
Merged

Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model#7425
tarekgh merged 3 commits into
dotnet:mainfrom
tarekgh:SupportByteLevelEncodingInBpeTokenizer

Conversation

@tarekgh

@tarekghtarekgh commented Mar 18, 2025•
edited
Loading

Copy link
Copy Markdown
Member

This change includes:

  • Adding support for ByteLevel encoding in the BPE tokenizer.
  • Introducing a new BpeTokenizer.Create method that allows creating a tokenizer using the newly introduced BpeOptions type. This enables users to obtain tokenizer data from various sources (e.g., tokenizer.json files) and instantiate the tokenizer using this new factory method.
  • Expanding test coverage to validate ByteLevel support.
  • Adding a test that loads the real tokenizer.json for the DeepSeek R1 model and verifies its functionality. Since the DeepSeek tokenizer utilizes ByteLevel encoding, it serves as an ideal test case for this feature.
  • Providing test code for loading tokenizer.json, which can be used as a template for loading other models' tokenizers when needed.
  • Introducing the CompositePreTokenizer to support the DeepSeek pre-tokenization scenario.

CopilotAI review requested due to automatic review settings March 18, 2025 21:51
CopilotAI reviewed Mar 18, 2025
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for ByteLevel encoding in the BPE tokenizer to enhance compatibility with the DeepSeek model. Key changes include:

  • Introducing a new BpeOptions type with a ByteLevel property.
  • Adding a CompositePreTokenizer implementation to apply multiple pre-tokenizers sequentially.
  • Refactoring and centralizing byte array appending logic in Helpers, along with updating the ToTokens method in Word to support mapping.

Reviewed Changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file
FileDescription
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdds a new options class for BPE tokenizers, including ByteLevel encoding support.
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csImplements a composite pre-tokenizer with special tokens handling.
src/Microsoft.ML.Tokenizers/Utils/Helpers.csIntroduces AppendToBytesArray to centralize byte transformation logic.
src/Microsoft.ML.Tokenizers/Model/Word.csUpdates the ToTokens method to incorporate an optional mapping parameter.
src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.csRefactors code to use the centralized Helpers.AppendToBytesArray method.
Files not reviewed (1)
  • eng/Versions.props: Language not supported

Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs Outdated
@tarekghtarekgh added this to the ML.NET 5.0 milestone Mar 18, 2025
@tarekgh
Update src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
5371dab
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@tarekgh

tarekgh commented Mar 18, 2025

Copy link
Copy Markdown
MemberAuthor

FYI @ericstj@luisquintanilla@stephentoub

@codecov

codecovBot commented Mar 18, 2025•
edited
Loading

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 75.08091% with 231 lines in your changes missing coverage. Please review.

Project coverage is 68.99%. Comparing base (adad40c) to head (39143c0).
Report is 1 commits behind head on main.

Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs62.13%96 Missing and 32 partials ⚠️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15%70 Missing and 6 partials ⚠️
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs94.59%13 Missing and 9 partials ⚠️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21%2 Missing and 1 partial ⚠️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs50.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7425 +/- ##
==========================================
+ Coverage 68.97% 68.99% +0.01% 
==========================================
Files 1481 1483 +2 Lines 273708 274563 +855 Branches 28285 28395 +110 ==========================================
+ Hits 188789 189431 +642 - Misses 77525 77694 +169 - Partials 7394 7438 +44
FlagCoverage Δ
Debug68.99% <75.08%> (+6.38%)⬆️
production63.26% <59.80%> (+0.65%)⬆️
test89.49% <94.59%> (∅)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/Word.cs63.80% <100.00%> (+2.38%)⬆️
src/Microsoft.ML.Tokenizers/Utils/Helpers.cs72.40% <100.00%> (+1.76%)⬆️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs72.65% <50.00%> (-0.11%)⬇️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21% <84.21%> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.31% <94.59%> (-2.69%)⬇️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <40.15%> (ø)
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <62.13%> (-5.03%)⬇️

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

michaelgsharp
michaelgsharp reviewed Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
/// <param name="byteLevel">Indicate whether to handle the input text in byte level.</param>
/// <param name="beginningOfSentenceToken">The beginning of sentence token.</param>
/// <param name="endOfSentenceToken">The end of sentence token.</param>
private BpeTokenizer(

@michaelgsharpmichaelgsharpMar 19, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be a good idea to have a constructor here that takes a BpeOptions so you don't have to change the signature everytime you have something like this?

@tarekghtarekghMar 19, 2025•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will require to have the existing BpeTokenizer.Create(..., ..., ...,..) to always create the options object to call that constructor. It may not be a big deal to do so but it will need some work to refactor that as we read the vocabs and merge differently in different cases.

michaelgsharp
michaelgsharp reviewed Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs Outdated
michaelgsharp
michaelgsharp reviewed Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs Outdated
michaelgsharp
michaelgsharp reviewed Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs Outdated
@tarekgh
tarekgh merged commit d4f690c into dotnet:mainMar 19, 2025
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Apr 19, 2025
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
+1 more reviewer
@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects

None yet

Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

3 participants

@tarekgh@michaelgsharp
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model - #7425
Merged
tarekgh merged 3 commits into
dotnet:mainfrom 
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Mar 19, 2025
Merged
Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model#7425
tarekgh merged 3 commits into
dotnet:mainfrom 
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Conversation
@tarekgh
@tarekghtarekgh
commented
Mar 18, 2025•

edited

Loading

Copy link


Copy Markdown

Member
This change includes:
  • Adding support for ByteLevel encoding in the BPE tokenizer.
  • Introducing a new BpeTokenizer.Create method that allows creating a tokenizer using the newly introduced BpeOptions type. This enables users to obtain tokenizer data from various sources (e.g., tokenizer.json files) and instantiate the tokenizer using this new factory method.
  • Expanding test coverage to validate ByteLevel support.
  • Adding a test that loads the real tokenizer.json for the DeepSeek R1 model and verifies its functionality. Since the DeepSeek tokenizer utilizes ByteLevel encoding, it serves as an ideal test case for this feature.
  • Providing test code for loading tokenizer.json, which can be used as a template for loading other models' tokenizers when needed.
  • Introducing the CompositePreTokenizer to support the DeepSeek pre-tokenization scenario.
CopilotAI
review requested due to automatic review settings
March 18, 2025 21:51
CopilotAI
reviewed
Mar 18, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This PR adds support for ByteLevel encoding in the BPE tokenizer to enhance compatibility with the DeepSeek model. Key changes include:
  • Introducing a new BpeOptions type with a ByteLevel property.
  • Adding a CompositePreTokenizer implementation to apply multiple pre-tokenizers sequentially.
  • Refactoring and centralizing byte array appending logic in Helpers, along with updating the ToTokens method in Word to support mapping.
Reviewed Changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
FileDescription
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdds a new options class for BPE tokenizers, including ByteLevel encoding support.
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csImplements a composite pre-tokenizer with special tokens handling.
src/Microsoft.ML.Tokenizers/Utils/Helpers.csIntroduces AppendToBytesArray to centralize byte transformation logic.
src/Microsoft.ML.Tokenizers/Model/Word.csUpdates the ToTokens method to incorporate an optional mapping parameter.
src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.csRefactors code to use the centralized Helpers.AppendToBytesArray method.
Files not reviewed (1)
  • eng/Versions.props: Language not supported
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekghtarekgh
added this to the ML.NET 5.0 milestone
Mar 18, 2025
@tarekgh
Update src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
5371dab
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@tarekgh
tarekgh
commented
Mar 18, 2025

Copy link


Copy Markdown

MemberAuthor
FYI @ericstj@luisquintanilla@stephentoub
@codecov
codecovBot
commented
Mar 18, 2025•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Attention: Patch coverage is 75.08091% with 231 lines in your changes missing coverage. Please review.
Project coverage is 68.99%. Comparing base (adad40c) to head (39143c0).

Report is 1 commits behind head on main.
Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs62.13%96 Missing and 32 partials ⚠️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15%70 Missing and 6 partials ⚠️
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs94.59%13 Missing and 9 partials ⚠️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21%2 Missing and 1 partial ⚠️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs50.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7425 +/- ##
==========================================
+ Coverage 68.97% 68.99% +0.01% 
==========================================
Files 1481 1483 +2 Lines 273708 274563 +855 Branches 28285 28395 +110 ==========================================
+ Hits 188789 189431 +642 - Misses 77525 77694 +169 - Partials 7394 7438 +44 
FlagCoverage Δ
Debug68.99% <75.08%> (+6.38%)⬆️
production63.26% <59.80%> (+0.65%)⬆️
test89.49% <94.59%> (∅)
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/Word.cs63.80% <100.00%> (+2.38%)⬆️
src/Microsoft.ML.Tokenizers/Utils/Helpers.cs72.40% <100.00%> (+1.76%)⬆️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs72.65% <50.00%> (-0.11%)⬇️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21% <84.21%> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.31% <94.59%> (-2.69%)⬇️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <40.15%> (ø)
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <62.13%> (-5.03%)⬇️
... and 7 files with indirect coverage changes
🚀 New features to boost your workflow: 
  • ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
 /// <param name="byteLevel">Indicate whether to handle the input text in byte level.</param>
 /// <param name="beginningOfSentenceToken">The beginning of sentence token.</param>
 /// <param name="endOfSentenceToken">The end of sentence token.</param>
 private BpeTokenizer(
@michaelgsharpmichaelgsharpMar 19, 2025

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be a good idea to have a constructor here that takes a BpeOptions so you don't have to change the signature everytime you have something like this?
@tarekghtarekghMar 19, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will require to have the existing BpeTokenizer.Create(..., ..., ...,..) to always create the options object to call that constructor. It may not be a big deal to do so but it will need some work to refactor that as we read the vocabs and merge differently in different cases.
michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
Outdated

michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
Outdated

michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekgh
tarekgh
merged commit d4f690c
into
dotnet:mainMar 19, 2025
@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Apr 19, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

+1 more reviewer

@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects


None yet


Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

3 participants

@tarekgh@michaelgsharp

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model - #7425
Merged
tarekgh merged 3 commits into
dotnet:mainfrom 
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Mar 19, 2025
Merged
Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model#7425
tarekgh merged 3 commits into
dotnet:mainfrom 
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Conversation
@tarekgh
@tarekghtarekgh
commented
Mar 18, 2025•

edited

Loading

Copy link


Copy Markdown

Member
This change includes:
  • Adding support for ByteLevel encoding in the BPE tokenizer.
  • Introducing a new BpeTokenizer.Create method that allows creating a tokenizer using the newly introduced BpeOptions type. This enables users to obtain tokenizer data from various sources (e.g., tokenizer.json files) and instantiate the tokenizer using this new factory method.
  • Expanding test coverage to validate ByteLevel support.
  • Adding a test that loads the real tokenizer.json for the DeepSeek R1 model and verifies its functionality. Since the DeepSeek tokenizer utilizes ByteLevel encoding, it serves as an ideal test case for this feature.
  • Providing test code for loading tokenizer.json, which can be used as a template for loading other models' tokenizers when needed.
  • Introducing the CompositePreTokenizer to support the DeepSeek pre-tokenization scenario.
CopilotAI
review requested due to automatic review settings
March 18, 2025 21:51
CopilotAI
reviewed
Mar 18, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This PR adds support for ByteLevel encoding in the BPE tokenizer to enhance compatibility with the DeepSeek model. Key changes include:
  • Introducing a new BpeOptions type with a ByteLevel property.
  • Adding a CompositePreTokenizer implementation to apply multiple pre-tokenizers sequentially.
  • Refactoring and centralizing byte array appending logic in Helpers, along with updating the ToTokens method in Word to support mapping.
Reviewed Changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
FileDescription
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdds a new options class for BPE tokenizers, including ByteLevel encoding support.
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csImplements a composite pre-tokenizer with special tokens handling.
src/Microsoft.ML.Tokenizers/Utils/Helpers.csIntroduces AppendToBytesArray to centralize byte transformation logic.
src/Microsoft.ML.Tokenizers/Model/Word.csUpdates the ToTokens method to incorporate an optional mapping parameter.
src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.csRefactors code to use the centralized Helpers.AppendToBytesArray method.
Files not reviewed (1)
  • eng/Versions.props: Language not supported
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekghtarekgh
added this to the ML.NET 5.0 milestone
Mar 18, 2025
@tarekgh
Update src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
5371dab
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@tarekgh
tarekgh
commented
Mar 18, 2025

Copy link


Copy Markdown

MemberAuthor
FYI @ericstj@luisquintanilla@stephentoub
@codecov
codecovBot
commented
Mar 18, 2025•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Attention: Patch coverage is 75.08091% with 231 lines in your changes missing coverage. Please review.
Project coverage is 68.99%. Comparing base (adad40c) to head (39143c0).

Report is 1 commits behind head on main.
Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs62.13%96 Missing and 32 partials ⚠️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15%70 Missing and 6 partials ⚠️
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs94.59%13 Missing and 9 partials ⚠️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21%2 Missing and 1 partial ⚠️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs50.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7425 +/- ##
==========================================
+ Coverage 68.97% 68.99% +0.01% 
==========================================
Files 1481 1483 +2 Lines 273708 274563 +855 Branches 28285 28395 +110 ==========================================
+ Hits 188789 189431 +642 - Misses 77525 77694 +169 - Partials 7394 7438 +44 
FlagCoverage Δ
Debug68.99% <75.08%> (+6.38%)⬆️
production63.26% <59.80%> (+0.65%)⬆️
test89.49% <94.59%> (∅)
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/Word.cs63.80% <100.00%> (+2.38%)⬆️
src/Microsoft.ML.Tokenizers/Utils/Helpers.cs72.40% <100.00%> (+1.76%)⬆️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs72.65% <50.00%> (-0.11%)⬇️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21% <84.21%> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.31% <94.59%> (-2.69%)⬇️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <40.15%> (ø)
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <62.13%> (-5.03%)⬇️
... and 7 files with indirect coverage changes
🚀 New features to boost your workflow: 
  • ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
 /// <param name="byteLevel">Indicate whether to handle the input text in byte level.</param>
 /// <param name="beginningOfSentenceToken">The beginning of sentence token.</param>
 /// <param name="endOfSentenceToken">The end of sentence token.</param>
 private BpeTokenizer(
@michaelgsharpmichaelgsharpMar 19, 2025

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be a good idea to have a constructor here that takes a BpeOptions so you don't have to change the signature everytime you have something like this?
@tarekghtarekghMar 19, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will require to have the existing BpeTokenizer.Create(..., ..., ...,..) to always create the options object to call that constructor. It may not be a big deal to do so but it will need some work to refactor that as we read the vocabs and merge differently in different cases.
michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
Outdated

michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
Outdated

michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekgh
tarekgh
merged commit d4f690c
into
dotnet:mainMar 19, 2025
@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Apr 19, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

+1 more reviewer

@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects


None yet


Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

3 participants

@tarekgh@michaelgsharp

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model - #7425
Merged
tarekgh merged 3 commits into
dotnet:mainfrom 
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Mar 19, 2025
Merged
Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model#7425
tarekgh merged 3 commits into
dotnet:mainfrom 
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Conversation
@tarekgh
@tarekghtarekgh
commented
Mar 18, 2025•

edited

Loading

Copy link


Copy Markdown

Member
This change includes:
  • Adding support for ByteLevel encoding in the BPE tokenizer.
  • Introducing a new BpeTokenizer.Create method that allows creating a tokenizer using the newly introduced BpeOptions type. This enables users to obtain tokenizer data from various sources (e.g., tokenizer.json files) and instantiate the tokenizer using this new factory method.
  • Expanding test coverage to validate ByteLevel support.
  • Adding a test that loads the real tokenizer.json for the DeepSeek R1 model and verifies its functionality. Since the DeepSeek tokenizer utilizes ByteLevel encoding, it serves as an ideal test case for this feature.
  • Providing test code for loading tokenizer.json, which can be used as a template for loading other models' tokenizers when needed.
  • Introducing the CompositePreTokenizer to support the DeepSeek pre-tokenization scenario.
CopilotAI
review requested due to automatic review settings
March 18, 2025 21:51
CopilotAI
reviewed
Mar 18, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This PR adds support for ByteLevel encoding in the BPE tokenizer to enhance compatibility with the DeepSeek model. Key changes include:
  • Introducing a new BpeOptions type with a ByteLevel property.
  • Adding a CompositePreTokenizer implementation to apply multiple pre-tokenizers sequentially.
  • Refactoring and centralizing byte array appending logic in Helpers, along with updating the ToTokens method in Word to support mapping.
Reviewed Changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
FileDescription
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdds a new options class for BPE tokenizers, including ByteLevel encoding support.
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csImplements a composite pre-tokenizer with special tokens handling.
src/Microsoft.ML.Tokenizers/Utils/Helpers.csIntroduces AppendToBytesArray to centralize byte transformation logic.
src/Microsoft.ML.Tokenizers/Model/Word.csUpdates the ToTokens method to incorporate an optional mapping parameter.
src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.csRefactors code to use the centralized Helpers.AppendToBytesArray method.
Files not reviewed (1)
  • eng/Versions.props: Language not supported
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekghtarekgh
added this to the ML.NET 5.0 milestone
Mar 18, 2025
@tarekgh
Update src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
5371dab
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@tarekgh
tarekgh
commented
Mar 18, 2025

Copy link


Copy Markdown

MemberAuthor
FYI @ericstj@luisquintanilla@stephentoub
@codecov
codecovBot
commented
Mar 18, 2025•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Attention: Patch coverage is 75.08091% with 231 lines in your changes missing coverage. Please review.
Project coverage is 68.99%. Comparing base (adad40c) to head (39143c0).

Report is 1 commits behind head on main.
Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs62.13%96 Missing and 32 partials ⚠️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15%70 Missing and 6 partials ⚠️
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs94.59%13 Missing and 9 partials ⚠️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21%2 Missing and 1 partial ⚠️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs50.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7425 +/- ##
==========================================
+ Coverage 68.97% 68.99% +0.01% 
==========================================
Files 1481 1483 +2 Lines 273708 274563 +855 Branches 28285 28395 +110 ==========================================
+ Hits 188789 189431 +642 - Misses 77525 77694 +169 - Partials 7394 7438 +44 
FlagCoverage Δ
Debug68.99% <75.08%> (+6.38%)⬆️
production63.26% <59.80%> (+0.65%)⬆️
test89.49% <94.59%> (∅)
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/Word.cs63.80% <100.00%> (+2.38%)⬆️
src/Microsoft.ML.Tokenizers/Utils/Helpers.cs72.40% <100.00%> (+1.76%)⬆️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs72.65% <50.00%> (-0.11%)⬇️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21% <84.21%> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.31% <94.59%> (-2.69%)⬇️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <40.15%> (ø)
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <62.13%> (-5.03%)⬇️
... and 7 files with indirect coverage changes
🚀 New features to boost your workflow: 
  • ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
 /// <param name="byteLevel">Indicate whether to handle the input text in byte level.</param>
 /// <param name="beginningOfSentenceToken">The beginning of sentence token.</param>
 /// <param name="endOfSentenceToken">The end of sentence token.</param>
 private BpeTokenizer(
@michaelgsharpmichaelgsharpMar 19, 2025

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be a good idea to have a constructor here that takes a BpeOptions so you don't have to change the signature everytime you have something like this?
@tarekghtarekghMar 19, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will require to have the existing BpeTokenizer.Create(..., ..., ...,..) to always create the options object to call that constructor. It may not be a big deal to do so but it will need some work to refactor that as we read the vocabs and merge differently in different cases.
michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
Outdated

michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
Outdated

michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekgh
tarekgh
merged commit d4f690c
into
dotnet:mainMar 19, 2025
@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Apr 19, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

+1 more reviewer

@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects


None yet


Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

3 participants

@tarekgh@michaelgsharp

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model - #7425
Merged
tarekgh merged 3 commits into
dotnet:mainfrom 
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Mar 19, 2025
Merged
Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model#7425
tarekgh merged 3 commits into
dotnet:mainfrom 
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Conversation
@tarekgh
@tarekghtarekgh
commented
Mar 18, 2025•

edited

Loading

Copy link


Copy Markdown

Member
This change includes:
  • Adding support for ByteLevel encoding in the BPE tokenizer.
  • Introducing a new BpeTokenizer.Create method that allows creating a tokenizer using the newly introduced BpeOptions type. This enables users to obtain tokenizer data from various sources (e.g., tokenizer.json files) and instantiate the tokenizer using this new factory method.
  • Expanding test coverage to validate ByteLevel support.
  • Adding a test that loads the real tokenizer.json for the DeepSeek R1 model and verifies its functionality. Since the DeepSeek tokenizer utilizes ByteLevel encoding, it serves as an ideal test case for this feature.
  • Providing test code for loading tokenizer.json, which can be used as a template for loading other models' tokenizers when needed.
  • Introducing the CompositePreTokenizer to support the DeepSeek pre-tokenization scenario.
CopilotAI
review requested due to automatic review settings
March 18, 2025 21:51
CopilotAI
reviewed
Mar 18, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This PR adds support for ByteLevel encoding in the BPE tokenizer to enhance compatibility with the DeepSeek model. Key changes include:
  • Introducing a new BpeOptions type with a ByteLevel property.
  • Adding a CompositePreTokenizer implementation to apply multiple pre-tokenizers sequentially.
  • Refactoring and centralizing byte array appending logic in Helpers, along with updating the ToTokens method in Word to support mapping.
Reviewed Changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
FileDescription
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdds a new options class for BPE tokenizers, including ByteLevel encoding support.
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csImplements a composite pre-tokenizer with special tokens handling.
src/Microsoft.ML.Tokenizers/Utils/Helpers.csIntroduces AppendToBytesArray to centralize byte transformation logic.
src/Microsoft.ML.Tokenizers/Model/Word.csUpdates the ToTokens method to incorporate an optional mapping parameter.
src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.csRefactors code to use the centralized Helpers.AppendToBytesArray method.
Files not reviewed (1)
  • eng/Versions.props: Language not supported
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekghtarekgh
added this to the ML.NET 5.0 milestone
Mar 18, 2025
@tarekgh
Update src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
5371dab
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@tarekgh
tarekgh
commented
Mar 18, 2025

Copy link


Copy Markdown

MemberAuthor
FYI @ericstj@luisquintanilla@stephentoub
@codecov
codecovBot
commented
Mar 18, 2025•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Attention: Patch coverage is 75.08091% with 231 lines in your changes missing coverage. Please review.
Project coverage is 68.99%. Comparing base (adad40c) to head (39143c0).

Report is 1 commits behind head on main.
Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs62.13%96 Missing and 32 partials ⚠️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15%70 Missing and 6 partials ⚠️
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs94.59%13 Missing and 9 partials ⚠️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21%2 Missing and 1 partial ⚠️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs50.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7425 +/- ##
==========================================
+ Coverage 68.97% 68.99% +0.01% 
==========================================
Files 1481 1483 +2 Lines 273708 274563 +855 Branches 28285 28395 +110 ==========================================
+ Hits 188789 189431 +642 - Misses 77525 77694 +169 - Partials 7394 7438 +44 
FlagCoverage Δ
Debug68.99% <75.08%> (+6.38%)⬆️
production63.26% <59.80%> (+0.65%)⬆️
test89.49% <94.59%> (∅)
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/Word.cs63.80% <100.00%> (+2.38%)⬆️
src/Microsoft.ML.Tokenizers/Utils/Helpers.cs72.40% <100.00%> (+1.76%)⬆️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs72.65% <50.00%> (-0.11%)⬇️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21% <84.21%> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.31% <94.59%> (-2.69%)⬇️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <40.15%> (ø)
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <62.13%> (-5.03%)⬇️
... and 7 files with indirect coverage changes
🚀 New features to boost your workflow: 
  • ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
 /// <param name="byteLevel">Indicate whether to handle the input text in byte level.</param>
 /// <param name="beginningOfSentenceToken">The beginning of sentence token.</param>
 /// <param name="endOfSentenceToken">The end of sentence token.</param>
 private BpeTokenizer(
@michaelgsharpmichaelgsharpMar 19, 2025

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be a good idea to have a constructor here that takes a BpeOptions so you don't have to change the signature everytime you have something like this?
@tarekghtarekghMar 19, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will require to have the existing BpeTokenizer.Create(..., ..., ...,..) to always create the options object to call that constructor. It may not be a big deal to do so but it will need some work to refactor that as we read the vocabs and merge differently in different cases.
michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
Outdated

michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
Outdated

michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekgh
tarekgh
merged commit d4f690c
into
dotnet:mainMar 19, 2025
@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Apr 19, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

+1 more reviewer

@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects


None yet


Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

3 participants

@tarekgh@michaelgsharp

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model - #7425
Merged
tarekgh merged 3 commits into
dotnet:mainfrom 
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Mar 19, 2025
Merged
Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model#7425
tarekgh merged 3 commits into
dotnet:mainfrom 
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Conversation
@tarekgh
@tarekghtarekgh
commented
Mar 18, 2025•

edited

Loading

Copy link


Copy Markdown

Member
This change includes:
  • Adding support for ByteLevel encoding in the BPE tokenizer.
  • Introducing a new BpeTokenizer.Create method that allows creating a tokenizer using the newly introduced BpeOptions type. This enables users to obtain tokenizer data from various sources (e.g., tokenizer.json files) and instantiate the tokenizer using this new factory method.
  • Expanding test coverage to validate ByteLevel support.
  • Adding a test that loads the real tokenizer.json for the DeepSeek R1 model and verifies its functionality. Since the DeepSeek tokenizer utilizes ByteLevel encoding, it serves as an ideal test case for this feature.
  • Providing test code for loading tokenizer.json, which can be used as a template for loading other models' tokenizers when needed.
  • Introducing the CompositePreTokenizer to support the DeepSeek pre-tokenization scenario.
CopilotAI
review requested due to automatic review settings
March 18, 2025 21:51
CopilotAI
reviewed
Mar 18, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This PR adds support for ByteLevel encoding in the BPE tokenizer to enhance compatibility with the DeepSeek model. Key changes include:
  • Introducing a new BpeOptions type with a ByteLevel property.
  • Adding a CompositePreTokenizer implementation to apply multiple pre-tokenizers sequentially.
  • Refactoring and centralizing byte array appending logic in Helpers, along with updating the ToTokens method in Word to support mapping.
Reviewed Changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
FileDescription
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdds a new options class for BPE tokenizers, including ByteLevel encoding support.
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csImplements a composite pre-tokenizer with special tokens handling.
src/Microsoft.ML.Tokenizers/Utils/Helpers.csIntroduces AppendToBytesArray to centralize byte transformation logic.
src/Microsoft.ML.Tokenizers/Model/Word.csUpdates the ToTokens method to incorporate an optional mapping parameter.
src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.csRefactors code to use the centralized Helpers.AppendToBytesArray method.
Files not reviewed (1)
  • eng/Versions.props: Language not supported
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekghtarekgh
added this to the ML.NET 5.0 milestone
Mar 18, 2025
@tarekgh
Update src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
5371dab
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@tarekgh
tarekgh
commented
Mar 18, 2025

Copy link


Copy Markdown

MemberAuthor
FYI @ericstj@luisquintanilla@stephentoub
@codecov
codecovBot
commented
Mar 18, 2025•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Attention: Patch coverage is 75.08091% with 231 lines in your changes missing coverage. Please review.
Project coverage is 68.99%. Comparing base (adad40c) to head (39143c0).

Report is 1 commits behind head on main.
Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs62.13%96 Missing and 32 partials ⚠️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15%70 Missing and 6 partials ⚠️
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs94.59%13 Missing and 9 partials ⚠️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21%2 Missing and 1 partial ⚠️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs50.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7425 +/- ##
==========================================
+ Coverage 68.97% 68.99% +0.01% 
==========================================
Files 1481 1483 +2 Lines 273708 274563 +855 Branches 28285 28395 +110 ==========================================
+ Hits 188789 189431 +642 - Misses 77525 77694 +169 - Partials 7394 7438 +44 
FlagCoverage Δ
Debug68.99% <75.08%> (+6.38%)⬆️
production63.26% <59.80%> (+0.65%)⬆️
test89.49% <94.59%> (∅)
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/Word.cs63.80% <100.00%> (+2.38%)⬆️
src/Microsoft.ML.Tokenizers/Utils/Helpers.cs72.40% <100.00%> (+1.76%)⬆️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs72.65% <50.00%> (-0.11%)⬇️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21% <84.21%> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.31% <94.59%> (-2.69%)⬇️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <40.15%> (ø)
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <62.13%> (-5.03%)⬇️
... and 7 files with indirect coverage changes
🚀 New features to boost your workflow: 
  • ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
 /// <param name="byteLevel">Indicate whether to handle the input text in byte level.</param>
 /// <param name="beginningOfSentenceToken">The beginning of sentence token.</param>
 /// <param name="endOfSentenceToken">The end of sentence token.</param>
 private BpeTokenizer(
@michaelgsharpmichaelgsharpMar 19, 2025

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be a good idea to have a constructor here that takes a BpeOptions so you don't have to change the signature everytime you have something like this?
@tarekghtarekghMar 19, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will require to have the existing BpeTokenizer.Create(..., ..., ...,..) to always create the options object to call that constructor. It may not be a big deal to do so but it will need some work to refactor that as we read the vocabs and merge differently in different cases.
michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
Outdated

michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
Outdated

michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekgh
tarekgh
merged commit d4f690c
into
dotnet:mainMar 19, 2025
@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Apr 19, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

+1 more reviewer

@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects


None yet


Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

3 participants

@tarekgh@michaelgsharp

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model - #7425
Merged
tarekgh merged 3 commits into
dotnet:mainfrom 
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Mar 19, 2025
Merged
Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model#7425
tarekgh merged 3 commits into
dotnet:mainfrom 
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Conversation
@tarekgh
@tarekghtarekgh
commented
Mar 18, 2025•

edited

Loading

Copy link


Copy Markdown

Member
This change includes:
  • Adding support for ByteLevel encoding in the BPE tokenizer.
  • Introducing a new BpeTokenizer.Create method that allows creating a tokenizer using the newly introduced BpeOptions type. This enables users to obtain tokenizer data from various sources (e.g., tokenizer.json files) and instantiate the tokenizer using this new factory method.
  • Expanding test coverage to validate ByteLevel support.
  • Adding a test that loads the real tokenizer.json for the DeepSeek R1 model and verifies its functionality. Since the DeepSeek tokenizer utilizes ByteLevel encoding, it serves as an ideal test case for this feature.
  • Providing test code for loading tokenizer.json, which can be used as a template for loading other models' tokenizers when needed.
  • Introducing the CompositePreTokenizer to support the DeepSeek pre-tokenization scenario.
CopilotAI
review requested due to automatic review settings
March 18, 2025 21:51
CopilotAI
reviewed
Mar 18, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This PR adds support for ByteLevel encoding in the BPE tokenizer to enhance compatibility with the DeepSeek model. Key changes include:
  • Introducing a new BpeOptions type with a ByteLevel property.
  • Adding a CompositePreTokenizer implementation to apply multiple pre-tokenizers sequentially.
  • Refactoring and centralizing byte array appending logic in Helpers, along with updating the ToTokens method in Word to support mapping.
Reviewed Changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
FileDescription
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdds a new options class for BPE tokenizers, including ByteLevel encoding support.
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csImplements a composite pre-tokenizer with special tokens handling.
src/Microsoft.ML.Tokenizers/Utils/Helpers.csIntroduces AppendToBytesArray to centralize byte transformation logic.
src/Microsoft.ML.Tokenizers/Model/Word.csUpdates the ToTokens method to incorporate an optional mapping parameter.
src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.csRefactors code to use the centralized Helpers.AppendToBytesArray method.
Files not reviewed (1)
  • eng/Versions.props: Language not supported
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekghtarekgh
added this to the ML.NET 5.0 milestone
Mar 18, 2025
@tarekgh
Update src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
5371dab
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@tarekgh
tarekgh
commented
Mar 18, 2025

Copy link


Copy Markdown

MemberAuthor
FYI @ericstj@luisquintanilla@stephentoub
@codecov
codecovBot
commented
Mar 18, 2025•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Attention: Patch coverage is 75.08091% with 231 lines in your changes missing coverage. Please review.
Project coverage is 68.99%. Comparing base (adad40c) to head (39143c0).

Report is 1 commits behind head on main.
Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs62.13%96 Missing and 32 partials ⚠️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15%70 Missing and 6 partials ⚠️
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs94.59%13 Missing and 9 partials ⚠️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21%2 Missing and 1 partial ⚠️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs50.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7425 +/- ##
==========================================
+ Coverage 68.97% 68.99% +0.01% 
==========================================
Files 1481 1483 +2 Lines 273708 274563 +855 Branches 28285 28395 +110 ==========================================
+ Hits 188789 189431 +642 - Misses 77525 77694 +169 - Partials 7394 7438 +44 
FlagCoverage Δ
Debug68.99% <75.08%> (+6.38%)⬆️
production63.26% <59.80%> (+0.65%)⬆️
test89.49% <94.59%> (∅)
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/Word.cs63.80% <100.00%> (+2.38%)⬆️
src/Microsoft.ML.Tokenizers/Utils/Helpers.cs72.40% <100.00%> (+1.76%)⬆️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs72.65% <50.00%> (-0.11%)⬇️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21% <84.21%> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.31% <94.59%> (-2.69%)⬇️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <40.15%> (ø)
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <62.13%> (-5.03%)⬇️
... and 7 files with indirect coverage changes
🚀 New features to boost your workflow: 
  • ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
 /// <param name="byteLevel">Indicate whether to handle the input text in byte level.</param>
 /// <param name="beginningOfSentenceToken">The beginning of sentence token.</param>
 /// <param name="endOfSentenceToken">The end of sentence token.</param>
 private BpeTokenizer(
@michaelgsharpmichaelgsharpMar 19, 2025

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be a good idea to have a constructor here that takes a BpeOptions so you don't have to change the signature everytime you have something like this?
@tarekghtarekghMar 19, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will require to have the existing BpeTokenizer.Create(..., ..., ...,..) to always create the options object to call that constructor. It may not be a big deal to do so but it will need some work to refactor that as we read the vocabs and merge differently in different cases.
michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
Outdated

michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
Outdated

michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekgh
tarekgh
merged commit d4f690c
into
dotnet:mainMar 19, 2025
@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Apr 19, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

+1 more reviewer

@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects


None yet


Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

3 participants

@tarekgh@michaelgsharp

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model - #7425
Merged
tarekgh merged 3 commits into
dotnet:mainfrom 
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Mar 19, 2025
Merged
Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model#7425
tarekgh merged 3 commits into
dotnet:mainfrom 
tarekgh:SupportByteLevelEncodingInBpeTokenizer
Conversation
@tarekgh
@tarekghtarekgh
commented
Mar 18, 2025•

edited

Loading

Copy link


Copy Markdown

Member
This change includes:
  • Adding support for ByteLevel encoding in the BPE tokenizer.
  • Introducing a new BpeTokenizer.Create method that allows creating a tokenizer using the newly introduced BpeOptions type. This enables users to obtain tokenizer data from various sources (e.g., tokenizer.json files) and instantiate the tokenizer using this new factory method.
  • Expanding test coverage to validate ByteLevel support.
  • Adding a test that loads the real tokenizer.json for the DeepSeek R1 model and verifies its functionality. Since the DeepSeek tokenizer utilizes ByteLevel encoding, it serves as an ideal test case for this feature.
  • Providing test code for loading tokenizer.json, which can be used as a template for loading other models' tokenizers when needed.
  • Introducing the CompositePreTokenizer to support the DeepSeek pre-tokenization scenario.
CopilotAI
review requested due to automatic review settings
March 18, 2025 21:51
CopilotAI
reviewed
Mar 18, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This PR adds support for ByteLevel encoding in the BPE tokenizer to enhance compatibility with the DeepSeek model. Key changes include:
  • Introducing a new BpeOptions type with a ByteLevel property.
  • Adding a CompositePreTokenizer implementation to apply multiple pre-tokenizers sequentially.
  • Refactoring and centralizing byte array appending logic in Helpers, along with updating the ToTokens method in Word to support mapping.
Reviewed Changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
FileDescription
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdds a new options class for BPE tokenizers, including ByteLevel encoding support.
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csImplements a composite pre-tokenizer with special tokens handling.
src/Microsoft.ML.Tokenizers/Utils/Helpers.csIntroduces AppendToBytesArray to centralize byte transformation logic.
src/Microsoft.ML.Tokenizers/Model/Word.csUpdates the ToTokens method to incorporate an optional mapping parameter.
src/Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.csRefactors code to use the centralized Helpers.AppendToBytesArray method.
Files not reviewed (1)
  • eng/Versions.props: Language not supported
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekghtarekgh
added this to the ML.NET 5.0 milestone
Mar 18, 2025
@tarekgh
Update src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
5371dab
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@tarekgh
tarekgh
commented
Mar 18, 2025

Copy link


Copy Markdown

MemberAuthor
FYI @ericstj@luisquintanilla@stephentoub
@codecov
codecovBot
commented
Mar 18, 2025•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Attention: Patch coverage is 75.08091% with 231 lines in your changes missing coverage. Please review.
Project coverage is 68.99%. Comparing base (adad40c) to head (39143c0).

Report is 1 commits behind head on main.
Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs62.13%96 Missing and 32 partials ⚠️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15%70 Missing and 6 partials ⚠️
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs94.59%13 Missing and 9 partials ⚠️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21%2 Missing and 1 partial ⚠️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs50.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7425 +/- ##
==========================================
+ Coverage 68.97% 68.99% +0.01% 
==========================================
Files 1481 1483 +2 Lines 273708 274563 +855 Branches 28285 28395 +110 ==========================================
+ Hits 188789 189431 +642 - Misses 77525 77694 +169 - Partials 7394 7438 +44 
FlagCoverage Δ
Debug68.99% <75.08%> (+6.38%)⬆️
production63.26% <59.80%> (+0.65%)⬆️
test89.49% <94.59%> (∅)
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/Word.cs63.80% <100.00%> (+2.38%)⬆️
src/Microsoft.ML.Tokenizers/Utils/Helpers.cs72.40% <100.00%> (+1.76%)⬆️
.../Microsoft.ML.Tokenizers/Model/CodeGenTokenizer.cs72.65% <50.00%> (-0.11%)⬇️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs84.21% <84.21%> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.31% <94.59%> (-2.69%)⬇️
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <40.15%> (ø)
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <62.13%> (-5.03%)⬇️
... and 7 files with indirect coverage changes
🚀 New features to boost your workflow: 
  • ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
 /// <param name="byteLevel">Indicate whether to handle the input text in byte level.</param>
 /// <param name="beginningOfSentenceToken">The beginning of sentence token.</param>
 /// <param name="endOfSentenceToken">The end of sentence token.</param>
 private BpeTokenizer(
@michaelgsharpmichaelgsharpMar 19, 2025

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be a good idea to have a constructor here that takes a BpeOptions so you don't have to change the signature everytime you have something like this?
@tarekghtarekghMar 19, 2025•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will require to have the existing BpeTokenizer.Create(..., ..., ...,..) to always create the options object to call that constructor. It may not be a big deal to do so but it will need some work to refactor that as we read the vocabs and merge differently in different cases.
michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
Outdated

michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs
Outdated

michaelgsharp
michaelgsharp
reviewed
Mar 19, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekgh
tarekgh
merged commit d4f690c
into
dotnet:mainMar 19, 2025
@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Apr 19, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

+1 more reviewer

@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects


None yet


Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

3 participants

@tarekgh@michaelgsharp