Address the design review feedback - #7513

Merged
ericstj merged 4 commits into
dotnet:mainfrom
tarekgh:AddressTheDesignReviewFeedback
Oct 1, 2025
Merged

Address the design review feedback#7513
ericstj merged 4 commits into
dotnet:mainfrom
tarekgh:AddressTheDesignReviewFeedback

Conversation

@tarekgh

@tarekghtarekgh commented Oct 1, 2025

Copy link
Copy Markdown
Member

Fixes#7512

CopilotAI review requested due to automatic review settings October 1, 2025 19:15
CopilotAI reviewed Oct 1, 2025
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR addresses design review feedback for the ML.NET tokenizers library by improving API consistency and usability. The changes focus on standardizing parameter types, adding constructor overloads, and improving property visibility.

  • Replaced tuple-based vocabulary parameters with KeyValuePair<string, int> for better API consistency
  • Added a new BpeOptions constructor that accepts file paths for vocabulary and merges files
  • Updated parameter naming and property visibility for better API design

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
FileDescription
test/Microsoft.ML.Tokenizers.Tests/BpeTests.csUpdated test code to use new KeyValuePair vocabulary format and added tests for new constructor
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csAdded missing namespace declaration
src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.csRenamed parameter from addBeginOfSentence to addBeginningOfSentence for consistency
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdded new constructor accepting file paths and changed vocabulary type from tuples to KeyValuePair
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.csUpdated to work with new vocabulary format and made BeginningOfSentenceToken property public

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs Outdated
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs Outdated
@tarekghtarekgh added this to the ML.NET 5.0 milestone Oct 1, 2025
@tarekghtarekgh self-assigned this Oct 1, 2025
@tarekgh
Apply suggestions from code review
628adb5
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
ericstj
ericstj reviewed Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
ericstj
ericstj reviewed Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs Outdated
ericstj
ericstj approved these changes Oct 1, 2025
View reviewed changes

@ericstjericstj left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple small questions, but code changes look fine.

tarekghand others added 2 commits October 1, 2025 12:45
@codecov

codecovBot commented Oct 1, 2025•
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 70.00000% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.00%. Comparing base (fb39755) to head (f80f3bf).
⚠️ Report is 4 commits behind head on main.

Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs56.09%12 Missing and 6 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7513 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 274048 +49 Branches 28258 28266 +8 ==========================================
+ Hits 189093 189113 +20 - Misses 77520 77540 +20 - Partials 7386 7395 +9
FlagCoverage Δ
Debug69.00% <70.00%> (-0.01%)⬇️
production63.29% <60.00%> (-0.01%)⬇️
test89.46% <100.00%> (+<0.01%)⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <100.00%> (ø)
...soft.ML.Tokenizers/Model/SentencePieceTokenizer.cs90.00% <100.00%> (ø)
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <ø> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.34% <100.00%> (+0.03%)⬆️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs63.15% <56.09%> (-21.06%)⬇️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ericstj
ericstj merged commit 965b755 into dotnet:mainOct 1, 2025
25 checks passed
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@ericstjericstjericstj approved these changes

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects

None yet

Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

Added Tokenizer's APIs for v2

3 participants

@tarekgh@ericstj
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Address the design review feedback - #7513
Merged
ericstj merged 4 commits into
dotnet:mainfrom 
tarekgh:AddressTheDesignReviewFeedback
Oct 1, 2025
Merged
Address the design review feedback#7513
ericstj merged 4 commits into
dotnet:mainfrom 
tarekgh:AddressTheDesignReviewFeedback
Conversation
@tarekgh
@tarekghtarekgh
commented
Oct 1, 2025

Copy link


Copy Markdown

Member
Fixes#7512
CopilotAI
review requested due to automatic review settings
October 1, 2025 19:15
CopilotAI
reviewed
Oct 1, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This PR addresses design review feedback for the ML.NET tokenizers library by improving API consistency and usability. The changes focus on standardizing parameter types, adding constructor overloads, and improving property visibility.
  • Replaced tuple-based vocabulary parameters with KeyValuePair<string, int> for better API consistency
  • Added a new BpeOptions constructor that accepts file paths for vocabulary and merges files
  • Updated parameter naming and property visibility for better API design
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
FileDescription
test/Microsoft.ML.Tokenizers.Tests/BpeTests.csUpdated test code to use new KeyValuePair vocabulary format and added tests for new constructor
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csAdded missing namespace declaration
src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.csRenamed parameter from addBeginOfSentence to addBeginningOfSentence for consistency
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdded new constructor accepting file paths and changed vocabulary type from tuples to KeyValuePair
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.csUpdated to work with new vocabulary format and made BeginningOfSentenceToken property public
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekghtarekgh
added this to the ML.NET 5.0 milestone
Oct 1, 2025
@tarekghtarekgh
self-assigned this
Oct 1, 2025
@tarekgh
Apply suggestions from code review
628adb5
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
ericstj
ericstj
reviewed
Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
ericstj
ericstj
reviewed
Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

ericstj
ericstj
approved these changes
Oct 1, 2025
View reviewed changes
@ericstjericstj
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple small questions, but code changes look fine.
tarekghand others
added 2 commits
October 1, 2025 12:45
@codecov
codecovBot
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

Codecov Report
❌ Patch coverage is 70.00000% with 18 lines in your changes missing coverage. Please review.

✅ Project coverage is 69.00%. Comparing base (fb39755) to head (f80f3bf).
⚠️ Report is 4 commits behind head on main.
Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs56.09%12 Missing and 6 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7513 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 274048 +49 Branches 28258 28266 +8 ==========================================
+ Hits 189093 189113 +20 - Misses 77520 77540 +20 - Partials 7386 7395 +9 
FlagCoverage Δ
Debug69.00% <70.00%> (-0.01%)⬇️
production63.29% <60.00%> (-0.01%)⬇️
test89.46% <100.00%> (+<0.01%)⬆️
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <100.00%> (ø)
...soft.ML.Tokenizers/Model/SentencePieceTokenizer.cs90.00% <100.00%> (ø)
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <ø> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.34% <100.00%> (+0.03%)⬆️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs63.15% <56.09%> (-21.06%)⬇️
... and 4 files with indirect coverage changes
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@ericstj
ericstj
merged commit 965b755
into
dotnet:mainOct 1, 2025

25 checks passed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@ericstjericstjericstj approved these changes

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects


None yet


Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

Added Tokenizer's APIs for v2


3 participants

@tarekgh@ericstj

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Address the design review feedback - #7513
Merged
ericstj merged 4 commits into
dotnet:mainfrom 
tarekgh:AddressTheDesignReviewFeedback
Oct 1, 2025
Merged
Address the design review feedback#7513
ericstj merged 4 commits into
dotnet:mainfrom 
tarekgh:AddressTheDesignReviewFeedback
Conversation
@tarekgh
@tarekghtarekgh
commented
Oct 1, 2025

Copy link


Copy Markdown

Member
Fixes#7512
CopilotAI
review requested due to automatic review settings
October 1, 2025 19:15
CopilotAI
reviewed
Oct 1, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This PR addresses design review feedback for the ML.NET tokenizers library by improving API consistency and usability. The changes focus on standardizing parameter types, adding constructor overloads, and improving property visibility.
  • Replaced tuple-based vocabulary parameters with KeyValuePair<string, int> for better API consistency
  • Added a new BpeOptions constructor that accepts file paths for vocabulary and merges files
  • Updated parameter naming and property visibility for better API design
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
FileDescription
test/Microsoft.ML.Tokenizers.Tests/BpeTests.csUpdated test code to use new KeyValuePair vocabulary format and added tests for new constructor
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csAdded missing namespace declaration
src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.csRenamed parameter from addBeginOfSentence to addBeginningOfSentence for consistency
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdded new constructor accepting file paths and changed vocabulary type from tuples to KeyValuePair
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.csUpdated to work with new vocabulary format and made BeginningOfSentenceToken property public
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekghtarekgh
added this to the ML.NET 5.0 milestone
Oct 1, 2025
@tarekghtarekgh
self-assigned this
Oct 1, 2025
@tarekgh
Apply suggestions from code review
628adb5
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
ericstj
ericstj
reviewed
Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
ericstj
ericstj
reviewed
Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

ericstj
ericstj
approved these changes
Oct 1, 2025
View reviewed changes
@ericstjericstj
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple small questions, but code changes look fine.
tarekghand others
added 2 commits
October 1, 2025 12:45
@codecov
codecovBot
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

Codecov Report
❌ Patch coverage is 70.00000% with 18 lines in your changes missing coverage. Please review.

✅ Project coverage is 69.00%. Comparing base (fb39755) to head (f80f3bf).
⚠️ Report is 4 commits behind head on main.
Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs56.09%12 Missing and 6 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7513 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 274048 +49 Branches 28258 28266 +8 ==========================================
+ Hits 189093 189113 +20 - Misses 77520 77540 +20 - Partials 7386 7395 +9 
FlagCoverage Δ
Debug69.00% <70.00%> (-0.01%)⬇️
production63.29% <60.00%> (-0.01%)⬇️
test89.46% <100.00%> (+<0.01%)⬆️
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <100.00%> (ø)
...soft.ML.Tokenizers/Model/SentencePieceTokenizer.cs90.00% <100.00%> (ø)
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <ø> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.34% <100.00%> (+0.03%)⬆️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs63.15% <56.09%> (-21.06%)⬇️
... and 4 files with indirect coverage changes
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@ericstj
ericstj
merged commit 965b755
into
dotnet:mainOct 1, 2025

25 checks passed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@ericstjericstjericstj approved these changes

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects


None yet


Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

Added Tokenizer's APIs for v2


3 participants

@tarekgh@ericstj

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Address the design review feedback - #7513
Merged
ericstj merged 4 commits into
dotnet:mainfrom 
tarekgh:AddressTheDesignReviewFeedback
Oct 1, 2025
Merged
Address the design review feedback#7513
ericstj merged 4 commits into
dotnet:mainfrom 
tarekgh:AddressTheDesignReviewFeedback
Conversation
@tarekgh
@tarekghtarekgh
commented
Oct 1, 2025

Copy link


Copy Markdown

Member
Fixes#7512
CopilotAI
review requested due to automatic review settings
October 1, 2025 19:15
CopilotAI
reviewed
Oct 1, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This PR addresses design review feedback for the ML.NET tokenizers library by improving API consistency and usability. The changes focus on standardizing parameter types, adding constructor overloads, and improving property visibility.
  • Replaced tuple-based vocabulary parameters with KeyValuePair<string, int> for better API consistency
  • Added a new BpeOptions constructor that accepts file paths for vocabulary and merges files
  • Updated parameter naming and property visibility for better API design
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
FileDescription
test/Microsoft.ML.Tokenizers.Tests/BpeTests.csUpdated test code to use new KeyValuePair vocabulary format and added tests for new constructor
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csAdded missing namespace declaration
src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.csRenamed parameter from addBeginOfSentence to addBeginningOfSentence for consistency
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdded new constructor accepting file paths and changed vocabulary type from tuples to KeyValuePair
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.csUpdated to work with new vocabulary format and made BeginningOfSentenceToken property public
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekghtarekgh
added this to the ML.NET 5.0 milestone
Oct 1, 2025
@tarekghtarekgh
self-assigned this
Oct 1, 2025
@tarekgh
Apply suggestions from code review
628adb5
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
ericstj
ericstj
reviewed
Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
ericstj
ericstj
reviewed
Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

ericstj
ericstj
approved these changes
Oct 1, 2025
View reviewed changes
@ericstjericstj
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple small questions, but code changes look fine.
tarekghand others
added 2 commits
October 1, 2025 12:45
@codecov
codecovBot
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

Codecov Report
❌ Patch coverage is 70.00000% with 18 lines in your changes missing coverage. Please review.

✅ Project coverage is 69.00%. Comparing base (fb39755) to head (f80f3bf).
⚠️ Report is 4 commits behind head on main.
Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs56.09%12 Missing and 6 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7513 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 274048 +49 Branches 28258 28266 +8 ==========================================
+ Hits 189093 189113 +20 - Misses 77520 77540 +20 - Partials 7386 7395 +9 
FlagCoverage Δ
Debug69.00% <70.00%> (-0.01%)⬇️
production63.29% <60.00%> (-0.01%)⬇️
test89.46% <100.00%> (+<0.01%)⬆️
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <100.00%> (ø)
...soft.ML.Tokenizers/Model/SentencePieceTokenizer.cs90.00% <100.00%> (ø)
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <ø> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.34% <100.00%> (+0.03%)⬆️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs63.15% <56.09%> (-21.06%)⬇️
... and 4 files with indirect coverage changes
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@ericstj
ericstj
merged commit 965b755
into
dotnet:mainOct 1, 2025

25 checks passed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@ericstjericstjericstj approved these changes

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects


None yet


Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

Added Tokenizer's APIs for v2


3 participants

@tarekgh@ericstj

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Address the design review feedback - #7513
Merged
ericstj merged 4 commits into
dotnet:mainfrom 
tarekgh:AddressTheDesignReviewFeedback
Oct 1, 2025
Merged
Address the design review feedback#7513
ericstj merged 4 commits into
dotnet:mainfrom 
tarekgh:AddressTheDesignReviewFeedback
Conversation
@tarekgh
@tarekghtarekgh
commented
Oct 1, 2025

Copy link


Copy Markdown

Member
Fixes#7512
CopilotAI
review requested due to automatic review settings
October 1, 2025 19:15
CopilotAI
reviewed
Oct 1, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This PR addresses design review feedback for the ML.NET tokenizers library by improving API consistency and usability. The changes focus on standardizing parameter types, adding constructor overloads, and improving property visibility.
  • Replaced tuple-based vocabulary parameters with KeyValuePair<string, int> for better API consistency
  • Added a new BpeOptions constructor that accepts file paths for vocabulary and merges files
  • Updated parameter naming and property visibility for better API design
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
FileDescription
test/Microsoft.ML.Tokenizers.Tests/BpeTests.csUpdated test code to use new KeyValuePair vocabulary format and added tests for new constructor
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csAdded missing namespace declaration
src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.csRenamed parameter from addBeginOfSentence to addBeginningOfSentence for consistency
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdded new constructor accepting file paths and changed vocabulary type from tuples to KeyValuePair
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.csUpdated to work with new vocabulary format and made BeginningOfSentenceToken property public
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekghtarekgh
added this to the ML.NET 5.0 milestone
Oct 1, 2025
@tarekghtarekgh
self-assigned this
Oct 1, 2025
@tarekgh
Apply suggestions from code review
628adb5
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
ericstj
ericstj
reviewed
Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
ericstj
ericstj
reviewed
Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

ericstj
ericstj
approved these changes
Oct 1, 2025
View reviewed changes
@ericstjericstj
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple small questions, but code changes look fine.
tarekghand others
added 2 commits
October 1, 2025 12:45
@codecov
codecovBot
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

Codecov Report
❌ Patch coverage is 70.00000% with 18 lines in your changes missing coverage. Please review.

✅ Project coverage is 69.00%. Comparing base (fb39755) to head (f80f3bf).
⚠️ Report is 4 commits behind head on main.
Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs56.09%12 Missing and 6 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7513 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 274048 +49 Branches 28258 28266 +8 ==========================================
+ Hits 189093 189113 +20 - Misses 77520 77540 +20 - Partials 7386 7395 +9 
FlagCoverage Δ
Debug69.00% <70.00%> (-0.01%)⬇️
production63.29% <60.00%> (-0.01%)⬇️
test89.46% <100.00%> (+<0.01%)⬆️
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <100.00%> (ø)
...soft.ML.Tokenizers/Model/SentencePieceTokenizer.cs90.00% <100.00%> (ø)
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <ø> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.34% <100.00%> (+0.03%)⬆️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs63.15% <56.09%> (-21.06%)⬇️
... and 4 files with indirect coverage changes
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@ericstj
ericstj
merged commit 965b755
into
dotnet:mainOct 1, 2025

25 checks passed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@ericstjericstjericstj approved these changes

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects


None yet


Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

Added Tokenizer's APIs for v2


3 participants

@tarekgh@ericstj

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Address the design review feedback - #7513
Merged
ericstj merged 4 commits into
dotnet:mainfrom 
tarekgh:AddressTheDesignReviewFeedback
Oct 1, 2025
Merged
Address the design review feedback#7513
ericstj merged 4 commits into
dotnet:mainfrom 
tarekgh:AddressTheDesignReviewFeedback
Conversation
@tarekgh
@tarekghtarekgh
commented
Oct 1, 2025

Copy link


Copy Markdown

Member
Fixes#7512
CopilotAI
review requested due to automatic review settings
October 1, 2025 19:15
CopilotAI
reviewed
Oct 1, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This PR addresses design review feedback for the ML.NET tokenizers library by improving API consistency and usability. The changes focus on standardizing parameter types, adding constructor overloads, and improving property visibility.
  • Replaced tuple-based vocabulary parameters with KeyValuePair<string, int> for better API consistency
  • Added a new BpeOptions constructor that accepts file paths for vocabulary and merges files
  • Updated parameter naming and property visibility for better API design
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
FileDescription
test/Microsoft.ML.Tokenizers.Tests/BpeTests.csUpdated test code to use new KeyValuePair vocabulary format and added tests for new constructor
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csAdded missing namespace declaration
src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.csRenamed parameter from addBeginOfSentence to addBeginningOfSentence for consistency
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdded new constructor accepting file paths and changed vocabulary type from tuples to KeyValuePair
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.csUpdated to work with new vocabulary format and made BeginningOfSentenceToken property public
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekghtarekgh
added this to the ML.NET 5.0 milestone
Oct 1, 2025
@tarekghtarekgh
self-assigned this
Oct 1, 2025
@tarekgh
Apply suggestions from code review
628adb5
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
ericstj
ericstj
reviewed
Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
ericstj
ericstj
reviewed
Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

ericstj
ericstj
approved these changes
Oct 1, 2025
View reviewed changes
@ericstjericstj
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple small questions, but code changes look fine.
tarekghand others
added 2 commits
October 1, 2025 12:45
@codecov
codecovBot
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

Codecov Report
❌ Patch coverage is 70.00000% with 18 lines in your changes missing coverage. Please review.

✅ Project coverage is 69.00%. Comparing base (fb39755) to head (f80f3bf).
⚠️ Report is 4 commits behind head on main.
Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs56.09%12 Missing and 6 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7513 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 274048 +49 Branches 28258 28266 +8 ==========================================
+ Hits 189093 189113 +20 - Misses 77520 77540 +20 - Partials 7386 7395 +9 
FlagCoverage Δ
Debug69.00% <70.00%> (-0.01%)⬇️
production63.29% <60.00%> (-0.01%)⬇️
test89.46% <100.00%> (+<0.01%)⬆️
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <100.00%> (ø)
...soft.ML.Tokenizers/Model/SentencePieceTokenizer.cs90.00% <100.00%> (ø)
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <ø> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.34% <100.00%> (+0.03%)⬆️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs63.15% <56.09%> (-21.06%)⬇️
... and 4 files with indirect coverage changes
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@ericstj
ericstj
merged commit 965b755
into
dotnet:mainOct 1, 2025

25 checks passed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@ericstjericstjericstj approved these changes

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects


None yet


Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

Added Tokenizer's APIs for v2


3 participants

@tarekgh@ericstj

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Address the design review feedback - #7513
Merged
ericstj merged 4 commits into
dotnet:mainfrom 
tarekgh:AddressTheDesignReviewFeedback
Oct 1, 2025
Merged
Address the design review feedback#7513
ericstj merged 4 commits into
dotnet:mainfrom 
tarekgh:AddressTheDesignReviewFeedback
Conversation
@tarekgh
@tarekghtarekgh
commented
Oct 1, 2025

Copy link


Copy Markdown

Member
Fixes#7512
CopilotAI
review requested due to automatic review settings
October 1, 2025 19:15
CopilotAI
reviewed
Oct 1, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This PR addresses design review feedback for the ML.NET tokenizers library by improving API consistency and usability. The changes focus on standardizing parameter types, adding constructor overloads, and improving property visibility.
  • Replaced tuple-based vocabulary parameters with KeyValuePair<string, int> for better API consistency
  • Added a new BpeOptions constructor that accepts file paths for vocabulary and merges files
  • Updated parameter naming and property visibility for better API design
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
FileDescription
test/Microsoft.ML.Tokenizers.Tests/BpeTests.csUpdated test code to use new KeyValuePair vocabulary format and added tests for new constructor
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csAdded missing namespace declaration
src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.csRenamed parameter from addBeginOfSentence to addBeginningOfSentence for consistency
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdded new constructor accepting file paths and changed vocabulary type from tuples to KeyValuePair
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.csUpdated to work with new vocabulary format and made BeginningOfSentenceToken property public
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekghtarekgh
added this to the ML.NET 5.0 milestone
Oct 1, 2025
@tarekghtarekgh
self-assigned this
Oct 1, 2025
@tarekgh
Apply suggestions from code review
628adb5
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
ericstj
ericstj
reviewed
Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
ericstj
ericstj
reviewed
Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

ericstj
ericstj
approved these changes
Oct 1, 2025
View reviewed changes
@ericstjericstj
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple small questions, but code changes look fine.
tarekghand others
added 2 commits
October 1, 2025 12:45
@codecov
codecovBot
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

Codecov Report
❌ Patch coverage is 70.00000% with 18 lines in your changes missing coverage. Please review.

✅ Project coverage is 69.00%. Comparing base (fb39755) to head (f80f3bf).
⚠️ Report is 4 commits behind head on main.
Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs56.09%12 Missing and 6 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7513 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 274048 +49 Branches 28258 28266 +8 ==========================================
+ Hits 189093 189113 +20 - Misses 77520 77540 +20 - Partials 7386 7395 +9 
FlagCoverage Δ
Debug69.00% <70.00%> (-0.01%)⬇️
production63.29% <60.00%> (-0.01%)⬇️
test89.46% <100.00%> (+<0.01%)⬆️
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <100.00%> (ø)
...soft.ML.Tokenizers/Model/SentencePieceTokenizer.cs90.00% <100.00%> (ø)
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <ø> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.34% <100.00%> (+0.03%)⬆️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs63.15% <56.09%> (-21.06%)⬇️
... and 4 files with indirect coverage changes
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@ericstj
ericstj
merged commit 965b755
into
dotnet:mainOct 1, 2025

25 checks passed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@ericstjericstjericstj approved these changes

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects


None yet


Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

Added Tokenizer's APIs for v2


3 participants

@tarekgh@ericstj

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Address the design review feedback - #7513
Merged
ericstj merged 4 commits into
dotnet:mainfrom 
tarekgh:AddressTheDesignReviewFeedback
Oct 1, 2025
Merged
Address the design review feedback#7513
ericstj merged 4 commits into
dotnet:mainfrom 
tarekgh:AddressTheDesignReviewFeedback
Conversation
@tarekgh
@tarekghtarekgh
commented
Oct 1, 2025

Copy link


Copy Markdown

Member
Fixes#7512
CopilotAI
review requested due to automatic review settings
October 1, 2025 19:15
CopilotAI
reviewed
Oct 1, 2025
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview
This PR addresses design review feedback for the ML.NET tokenizers library by improving API consistency and usability. The changes focus on standardizing parameter types, adding constructor overloads, and improving property visibility.
  • Replaced tuple-based vocabulary parameters with KeyValuePair<string, int> for better API consistency
  • Added a new BpeOptions constructor that accepts file paths for vocabulary and merges files
  • Updated parameter naming and property visibility for better API design
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
FileDescription
test/Microsoft.ML.Tokenizers.Tests/BpeTests.csUpdated test code to use new KeyValuePair vocabulary format and added tests for new constructor
src/Microsoft.ML.Tokenizers/PreTokenizer/CompositePreTokenizer.csAdded missing namespace declaration
src/Microsoft.ML.Tokenizers/Model/SentencePieceTokenizer.csRenamed parameter from addBeginOfSentence to addBeginningOfSentence for consistency
src/Microsoft.ML.Tokenizers/Model/BpeOptions.csAdded new constructor accepting file paths and changed vocabulary type from tuples to KeyValuePair
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.csUpdated to work with new vocabulary format and made BeginningOfSentenceToken property public
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

@tarekghtarekgh
added this to the ML.NET 5.0 milestone
Oct 1, 2025
@tarekghtarekgh
self-assigned this
Oct 1, 2025
@tarekgh
Apply suggestions from code review
628adb5
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
ericstj
ericstj
reviewed
Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
ericstj
ericstj
reviewed
Oct 1, 2025
View reviewed changes
Comment threadsrc/Microsoft.ML.Tokenizers/Model/BpeOptions.cs
Outdated

ericstj
ericstj
approved these changes
Oct 1, 2025
View reviewed changes
@ericstjericstj
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple small questions, but code changes look fine.
tarekghand others
added 2 commits
October 1, 2025 12:45
@codecov
codecovBot
commented
Oct 1, 2025•

edited

Loading

Copy link


Copy Markdown

Codecov Report
❌ Patch coverage is 70.00000% with 18 lines in your changes missing coverage. Please review.

✅ Project coverage is 69.00%. Comparing base (fb39755) to head (f80f3bf).
⚠️ Report is 4 commits behind head on main.
Files with missing linesPatch %Lines
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs56.09%12 Missing and 6 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #7513 +/- ##
==========================================
- Coverage 69.01% 69.00% -0.01% 
==========================================
Files 1482 1482 Lines 273999 274048 +49 Branches 28258 28266 +8 ==========================================
+ Hits 189093 189113 +20 - Misses 77520 77540 +20 - Partials 7386 7395 +9 
FlagCoverage Δ
Debug69.00% <70.00%> (-0.01%)⬇️
production63.29% <60.00%> (-0.01%)⬇️
test89.46% <100.00%> (+<0.01%)⬆️
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
src/Microsoft.ML.Tokenizers/Model/BPETokenizer.cs71.98% <100.00%> (ø)
...soft.ML.Tokenizers/Model/SentencePieceTokenizer.cs90.00% <100.00%> (ø)
...L.Tokenizers/PreTokenizer/CompositePreTokenizer.cs40.15% <ø> (ø)
test/Microsoft.ML.Tokenizers.Tests/BpeTests.cs97.34% <100.00%> (+0.03%)⬆️
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs63.15% <56.09%> (-21.06%)⬇️
... and 4 files with indirect coverage changes
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@ericstj
ericstj
merged commit 965b755
into
dotnet:mainOct 1, 2025

25 checks passed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Nov 1, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@ericstjericstjericstj approved these changes

Assignees

@tarekghtarekgh

Labels

Tokenizers

Projects


None yet


Milestone

ML.NET 5.0

Development

Successfully merging this pull request may close these issues.

Added Tokenizer's APIs for v2


3 participants

@tarekgh@ericstj