Add gpt-5.3 model support to TiktokenTokenizer - #7579

Merged
tarekgh merged 2 commits into
mainfrom
copilot/add-gpt-5-3-support
Feb 13, 2026
Merged

Add gpt-5.3 model support to TiktokenTokenizer#7579
tarekgh merged 2 commits into
mainfrom
copilot/add-gpt-5-3-support

Conversation

CopilotAI commented Feb 12, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Adds support for the gpt-5.3 model family to TiktokenTokenizer using O200kBase encoding.

Changes

  • TiktokenTokenizer.cs: Added gpt-5.3 and gpt-5.3- mappings to model encoding dictionaries
  • TiktokenTests.cs: Added test coverage for gpt-5.3 base and mini variants

Usage

vartokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3");vartokens=tokenizer.EncodeToIds("Hello, world!");// Also supports variantsvarminiTokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3-mini");

Follows the same pattern established for gpt-5.2 support.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • dnceng.pkgs.visualstudio.com
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet dotnet test test/Microsoft.ML.Tokenizers.Tests/Microsoft.ML.Tokenizers.Tests.csproj --filter FullyQualifiedName~TiktokenTests --no-build (dns block)
  • openaipublic.blob.core.windows.net
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.deps.json /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/testhost.dll --port 45083 --endpoint 127.0.0.1:045083 --role client --parentprocessid 6859 --telemetryoptedin false (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

@stephentoub
Add gpt-5.3 support in TiktokenTokenizer
cff6f1e
Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
CopilotAI changed the title [WIP] Add gpt-5.3 support similar to gpt-5.2Add gpt-5.3 model support to TiktokenTokenizerFeb 12, 2026
@stephentoub
stephentoub marked this pull request as ready for review February 12, 2026 22:13
CopilotAI reviewed Feb 12, 2026
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request adds support for the GPT-5.3 model family to the TiktokenTokenizer, following the established pattern used for previous GPT model versions (gpt-5.2, gpt-5.1, etc.). The changes enable tokenization for both the base "gpt-5.3" model and its variants (e.g., "gpt-5.3-mini") using the O200kBase encoding.

Changes:

  • Added gpt-5.3 model mappings to TiktokenTokenizer for both prefix and exact matching
  • Added GPT5_3 static tokenizer property and comprehensive test coverage for the new model
  • Included test cases for both base and mini variants in the test suite

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

FileDescription
src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.csAdded "gpt-5.3-" prefix mapping and "gpt-5.3" exact name mapping to model encoding arrays, both using O200kBase encoding
test/Microsoft.ML.Tokenizers.Tests/TiktokenTests.csAdded GPT5_3 static property and test data entries for gpt-5.3 and gpt-5.3-mini variants in TestAllSupportedModelNames and TestCreationUsingModel methods

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tarekgh
tarekgh approved these changes Feb 12, 2026
View reviewed changes

@tarekghtarekgh left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@codecov

codecovBot commented Feb 12, 2026•
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.06%. Comparing base (3604580) to head (cff6f1e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@ Coverage Diff @@## main #7579 +/- ##
=======================================
Coverage 69.05% 69.06% =======================================
Files 1483 1483 Lines 274362 274365 +3 Branches 28270 28270 =======================================
+ Hits 189466 189482 +16 + Misses 77510 77498 -12 + Partials 7386 7385 -1
FlagCoverage Δ
Debug69.06% <100.00%> (+<0.01%)⬆️
production63.32% <100.00%> (+<0.01%)⬆️
test89.51% <100.00%> (+<0.01%)⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing linesCoverage Δ
...Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs80.00% <100.00%> (+0.04%)⬆️
...est/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs99.09% <100.00%> (+<0.01%)⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tarekgh
tarekgh enabled auto-merge (squash) February 13, 2026 00:02
@tarekgh
tarekgh merged commit 989f5a0 into mainFeb 13, 2026
25 of 31 checks passed
This was referenced Feb 13, 2026
Open
Open
Closed
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Mar 15, 2026
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@tarekghtarekghtarekgh approved these changes
+1 more reviewer
@stephentoubstephentoubstephentoub approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@stephentoubstephentoub

Copilot code reviewCopilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@stephentoub@tarekgh
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Add gpt-5.3 model support to TiktokenTokenizer - #7579
Merged
tarekgh merged 2 commits into
mainfrom 
copilot/add-gpt-5-3-support
Feb 13, 2026
Merged
Add gpt-5.3 model support to TiktokenTokenizer#7579
tarekgh merged 2 commits into
mainfrom 
copilot/add-gpt-5-3-support
Conversation
CopilotAI
commented
Feb 12, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Adds support for the gpt-5.3 model family to TiktokenTokenizer using O200kBase encoding.
Changes
  • TiktokenTokenizer.cs: Added gpt-5.3 and gpt-5.3- mappings to model encoding dictionaries
  • TiktokenTests.cs: Added test coverage for gpt-5.3 base and mini variants
Usage
vartokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3");vartokens=tokenizer.EncodeToIds("Hello, world!");// Also supports variantsvarminiTokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3-mini");
Follows the same pattern established for gpt-5.2 support.
Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
  • dnceng.pkgs.visualstudio.com
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet dotnet test test/Microsoft.ML.Tokenizers.Tests/Microsoft.ML.Tokenizers.Tests.csproj --filter FullyQualifiedName~TiktokenTests --no-build (dns block)
  • openaipublic.blob.core.windows.net
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.deps.json /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/testhost.dll --port 45083 --endpoint 127.0.0.1:045083 --role client --parentprocessid 6859 --telemetryoptedin false (dns block)
If you need me to access, download, or install something from one of these locations, you can either:
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.
@stephentoub
Add gpt-5.3 support in TiktokenTokenizer
cff6f1e
Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
CopilotAI
changed the title
[WIP] Add gpt-5.3 support similar to gpt-5.2Add gpt-5.3 model support to TiktokenTokenizerFeb 12, 2026
@stephentoub
stephentoub
marked this pull request as ready for review
February 12, 2026 22:13
CopilotAI
reviewed
Feb 12, 2026
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview
This pull request adds support for the GPT-5.3 model family to the TiktokenTokenizer, following the established pattern used for previous GPT model versions (gpt-5.2, gpt-5.1, etc.). The changes enable tokenization for both the base "gpt-5.3" model and its variants (e.g., "gpt-5.3-mini") using the O200kBase encoding.
Changes:
  • Added gpt-5.3 model mappings to TiktokenTokenizer for both prefix and exact matching
  • Added GPT5_3 static tokenizer property and comprehensive test coverage for the new model
  • Included test cases for both base and mini variants in the test suite
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
FileDescription
src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.csAdded "gpt-5.3-" prefix mapping and "gpt-5.3" exact name mapping to model encoding arrays, both using O200kBase encoding
test/Microsoft.ML.Tokenizers.Tests/TiktokenTests.csAdded GPT5_3 static property and test data entries for gpt-5.3 and gpt-5.3-mini variants in TestAllSupportedModelNames and TestCreationUsingModel methods
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tarekgh
tarekgh
approved these changes
Feb 12, 2026
View reviewed changes
@tarekghtarekgh
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
@codecov
codecovBot
commented
Feb 12, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
✅ All modified and coverable lines are covered by tests.

✅ Project coverage is 69.06%. Comparing base (3604580) to head (cff6f1e).
⚠️ Report is 1 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@## main #7579 +/- ##
=======================================
Coverage 69.05% 69.06% =======================================
Files 1483 1483 Lines 274362 274365 +3 Branches 28270 28270 =======================================
+ Hits 189466 189482 +16 + Misses 77510 77498 -12 + Partials 7386 7385 -1 
FlagCoverage Δ
Debug69.06% <100.00%> (+<0.01%)⬆️
production63.32% <100.00%> (+<0.01%)⬆️
test89.51% <100.00%> (+<0.01%)⬆️
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
...Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs80.00% <100.00%> (+0.04%)⬆️
...est/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs99.09% <100.00%> (+<0.01%)⬆️
... and 3 files with indirect coverage changes
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@tarekgh
tarekgh
enabled auto-merge (squash)
February 13, 2026 00:02
@tarekgh
tarekgh
merged commit 989f5a0
into
mainFeb 13, 2026

25 of 31 checks passed


This was referenced Feb 13, 2026

Open


Open


Closed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Mar 15, 2026
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@tarekghtarekghtarekgh approved these changes

+1 more reviewer

@stephentoubstephentoubstephentoub approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@stephentoubstephentoub
Copilot code reviewCopilot

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@stephentoub@tarekgh

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add gpt-5.3 model support to TiktokenTokenizer - #7579
Merged
tarekgh merged 2 commits into
mainfrom 
copilot/add-gpt-5-3-support
Feb 13, 2026
Merged
Add gpt-5.3 model support to TiktokenTokenizer#7579
tarekgh merged 2 commits into
mainfrom 
copilot/add-gpt-5-3-support
Conversation
CopilotAI
commented
Feb 12, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Adds support for the gpt-5.3 model family to TiktokenTokenizer using O200kBase encoding.
Changes
  • TiktokenTokenizer.cs: Added gpt-5.3 and gpt-5.3- mappings to model encoding dictionaries
  • TiktokenTests.cs: Added test coverage for gpt-5.3 base and mini variants
Usage
vartokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3");vartokens=tokenizer.EncodeToIds("Hello, world!");// Also supports variantsvarminiTokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3-mini");
Follows the same pattern established for gpt-5.2 support.
Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
  • dnceng.pkgs.visualstudio.com
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet dotnet test test/Microsoft.ML.Tokenizers.Tests/Microsoft.ML.Tokenizers.Tests.csproj --filter FullyQualifiedName~TiktokenTests --no-build (dns block)
  • openaipublic.blob.core.windows.net
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.deps.json /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/testhost.dll --port 45083 --endpoint 127.0.0.1:045083 --role client --parentprocessid 6859 --telemetryoptedin false (dns block)
If you need me to access, download, or install something from one of these locations, you can either:
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.
@stephentoub
Add gpt-5.3 support in TiktokenTokenizer
cff6f1e
Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
CopilotAI
changed the title
[WIP] Add gpt-5.3 support similar to gpt-5.2Add gpt-5.3 model support to TiktokenTokenizerFeb 12, 2026
@stephentoub
stephentoub
marked this pull request as ready for review
February 12, 2026 22:13
CopilotAI
reviewed
Feb 12, 2026
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview
This pull request adds support for the GPT-5.3 model family to the TiktokenTokenizer, following the established pattern used for previous GPT model versions (gpt-5.2, gpt-5.1, etc.). The changes enable tokenization for both the base "gpt-5.3" model and its variants (e.g., "gpt-5.3-mini") using the O200kBase encoding.
Changes:
  • Added gpt-5.3 model mappings to TiktokenTokenizer for both prefix and exact matching
  • Added GPT5_3 static tokenizer property and comprehensive test coverage for the new model
  • Included test cases for both base and mini variants in the test suite
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
FileDescription
src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.csAdded "gpt-5.3-" prefix mapping and "gpt-5.3" exact name mapping to model encoding arrays, both using O200kBase encoding
test/Microsoft.ML.Tokenizers.Tests/TiktokenTests.csAdded GPT5_3 static property and test data entries for gpt-5.3 and gpt-5.3-mini variants in TestAllSupportedModelNames and TestCreationUsingModel methods
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tarekgh
tarekgh
approved these changes
Feb 12, 2026
View reviewed changes
@tarekghtarekgh
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
@codecov
codecovBot
commented
Feb 12, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
✅ All modified and coverable lines are covered by tests.

✅ Project coverage is 69.06%. Comparing base (3604580) to head (cff6f1e).
⚠️ Report is 1 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@## main #7579 +/- ##
=======================================
Coverage 69.05% 69.06% =======================================
Files 1483 1483 Lines 274362 274365 +3 Branches 28270 28270 =======================================
+ Hits 189466 189482 +16 + Misses 77510 77498 -12 + Partials 7386 7385 -1 
FlagCoverage Δ
Debug69.06% <100.00%> (+<0.01%)⬆️
production63.32% <100.00%> (+<0.01%)⬆️
test89.51% <100.00%> (+<0.01%)⬆️
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
...Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs80.00% <100.00%> (+0.04%)⬆️
...est/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs99.09% <100.00%> (+<0.01%)⬆️
... and 3 files with indirect coverage changes
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@tarekgh
tarekgh
enabled auto-merge (squash)
February 13, 2026 00:02
@tarekgh
tarekgh
merged commit 989f5a0
into
mainFeb 13, 2026

25 of 31 checks passed


This was referenced Feb 13, 2026

Open


Open


Closed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Mar 15, 2026
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@tarekghtarekghtarekgh approved these changes

+1 more reviewer

@stephentoubstephentoubstephentoub approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@stephentoubstephentoub
Copilot code reviewCopilot

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@stephentoub@tarekgh

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add gpt-5.3 model support to TiktokenTokenizer - #7579
Merged
tarekgh merged 2 commits into
mainfrom 
copilot/add-gpt-5-3-support
Feb 13, 2026
Merged
Add gpt-5.3 model support to TiktokenTokenizer#7579
tarekgh merged 2 commits into
mainfrom 
copilot/add-gpt-5-3-support
Conversation
CopilotAI
commented
Feb 12, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Adds support for the gpt-5.3 model family to TiktokenTokenizer using O200kBase encoding.
Changes
  • TiktokenTokenizer.cs: Added gpt-5.3 and gpt-5.3- mappings to model encoding dictionaries
  • TiktokenTests.cs: Added test coverage for gpt-5.3 base and mini variants
Usage
vartokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3");vartokens=tokenizer.EncodeToIds("Hello, world!");// Also supports variantsvarminiTokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3-mini");
Follows the same pattern established for gpt-5.2 support.
Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
  • dnceng.pkgs.visualstudio.com
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet dotnet test test/Microsoft.ML.Tokenizers.Tests/Microsoft.ML.Tokenizers.Tests.csproj --filter FullyQualifiedName~TiktokenTests --no-build (dns block)
  • openaipublic.blob.core.windows.net
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.deps.json /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/testhost.dll --port 45083 --endpoint 127.0.0.1:045083 --role client --parentprocessid 6859 --telemetryoptedin false (dns block)
If you need me to access, download, or install something from one of these locations, you can either:
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.
@stephentoub
Add gpt-5.3 support in TiktokenTokenizer
cff6f1e
Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
CopilotAI
changed the title
[WIP] Add gpt-5.3 support similar to gpt-5.2Add gpt-5.3 model support to TiktokenTokenizerFeb 12, 2026
@stephentoub
stephentoub
marked this pull request as ready for review
February 12, 2026 22:13
CopilotAI
reviewed
Feb 12, 2026
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview
This pull request adds support for the GPT-5.3 model family to the TiktokenTokenizer, following the established pattern used for previous GPT model versions (gpt-5.2, gpt-5.1, etc.). The changes enable tokenization for both the base "gpt-5.3" model and its variants (e.g., "gpt-5.3-mini") using the O200kBase encoding.
Changes:
  • Added gpt-5.3 model mappings to TiktokenTokenizer for both prefix and exact matching
  • Added GPT5_3 static tokenizer property and comprehensive test coverage for the new model
  • Included test cases for both base and mini variants in the test suite
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
FileDescription
src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.csAdded "gpt-5.3-" prefix mapping and "gpt-5.3" exact name mapping to model encoding arrays, both using O200kBase encoding
test/Microsoft.ML.Tokenizers.Tests/TiktokenTests.csAdded GPT5_3 static property and test data entries for gpt-5.3 and gpt-5.3-mini variants in TestAllSupportedModelNames and TestCreationUsingModel methods
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tarekgh
tarekgh
approved these changes
Feb 12, 2026
View reviewed changes
@tarekghtarekgh
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
@codecov
codecovBot
commented
Feb 12, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
✅ All modified and coverable lines are covered by tests.

✅ Project coverage is 69.06%. Comparing base (3604580) to head (cff6f1e).
⚠️ Report is 1 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@## main #7579 +/- ##
=======================================
Coverage 69.05% 69.06% =======================================
Files 1483 1483 Lines 274362 274365 +3 Branches 28270 28270 =======================================
+ Hits 189466 189482 +16 + Misses 77510 77498 -12 + Partials 7386 7385 -1 
FlagCoverage Δ
Debug69.06% <100.00%> (+<0.01%)⬆️
production63.32% <100.00%> (+<0.01%)⬆️
test89.51% <100.00%> (+<0.01%)⬆️
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
...Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs80.00% <100.00%> (+0.04%)⬆️
...est/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs99.09% <100.00%> (+<0.01%)⬆️
... and 3 files with indirect coverage changes
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@tarekgh
tarekgh
enabled auto-merge (squash)
February 13, 2026 00:02
@tarekgh
tarekgh
merged commit 989f5a0
into
mainFeb 13, 2026

25 of 31 checks passed


This was referenced Feb 13, 2026

Open


Open


Closed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Mar 15, 2026
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@tarekghtarekghtarekgh approved these changes

+1 more reviewer

@stephentoubstephentoubstephentoub approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@stephentoubstephentoub
Copilot code reviewCopilot

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@stephentoub@tarekgh

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Add gpt-5.3 model support to TiktokenTokenizer - #7579
Merged
tarekgh merged 2 commits into
mainfrom 
copilot/add-gpt-5-3-support
Feb 13, 2026
Merged
Add gpt-5.3 model support to TiktokenTokenizer#7579
tarekgh merged 2 commits into
mainfrom 
copilot/add-gpt-5-3-support
Conversation
CopilotAI
commented
Feb 12, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Adds support for the gpt-5.3 model family to TiktokenTokenizer using O200kBase encoding.
Changes
  • TiktokenTokenizer.cs: Added gpt-5.3 and gpt-5.3- mappings to model encoding dictionaries
  • TiktokenTests.cs: Added test coverage for gpt-5.3 base and mini variants
Usage
vartokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3");vartokens=tokenizer.EncodeToIds("Hello, world!");// Also supports variantsvarminiTokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3-mini");
Follows the same pattern established for gpt-5.2 support.
Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
  • dnceng.pkgs.visualstudio.com
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet dotnet test test/Microsoft.ML.Tokenizers.Tests/Microsoft.ML.Tokenizers.Tests.csproj --filter FullyQualifiedName~TiktokenTests --no-build (dns block)
  • openaipublic.blob.core.windows.net
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.deps.json /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/testhost.dll --port 45083 --endpoint 127.0.0.1:045083 --role client --parentprocessid 6859 --telemetryoptedin false (dns block)
If you need me to access, download, or install something from one of these locations, you can either:
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.
@stephentoub
Add gpt-5.3 support in TiktokenTokenizer
cff6f1e
Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
CopilotAI
changed the title
[WIP] Add gpt-5.3 support similar to gpt-5.2Add gpt-5.3 model support to TiktokenTokenizerFeb 12, 2026
@stephentoub
stephentoub
marked this pull request as ready for review
February 12, 2026 22:13
CopilotAI
reviewed
Feb 12, 2026
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview
This pull request adds support for the GPT-5.3 model family to the TiktokenTokenizer, following the established pattern used for previous GPT model versions (gpt-5.2, gpt-5.1, etc.). The changes enable tokenization for both the base "gpt-5.3" model and its variants (e.g., "gpt-5.3-mini") using the O200kBase encoding.
Changes:
  • Added gpt-5.3 model mappings to TiktokenTokenizer for both prefix and exact matching
  • Added GPT5_3 static tokenizer property and comprehensive test coverage for the new model
  • Included test cases for both base and mini variants in the test suite
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
FileDescription
src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.csAdded "gpt-5.3-" prefix mapping and "gpt-5.3" exact name mapping to model encoding arrays, both using O200kBase encoding
test/Microsoft.ML.Tokenizers.Tests/TiktokenTests.csAdded GPT5_3 static property and test data entries for gpt-5.3 and gpt-5.3-mini variants in TestAllSupportedModelNames and TestCreationUsingModel methods
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tarekgh
tarekgh
approved these changes
Feb 12, 2026
View reviewed changes
@tarekghtarekgh
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
@codecov
codecovBot
commented
Feb 12, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
✅ All modified and coverable lines are covered by tests.

✅ Project coverage is 69.06%. Comparing base (3604580) to head (cff6f1e).
⚠️ Report is 1 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@## main #7579 +/- ##
=======================================
Coverage 69.05% 69.06% =======================================
Files 1483 1483 Lines 274362 274365 +3 Branches 28270 28270 =======================================
+ Hits 189466 189482 +16 + Misses 77510 77498 -12 + Partials 7386 7385 -1 
FlagCoverage Δ
Debug69.06% <100.00%> (+<0.01%)⬆️
production63.32% <100.00%> (+<0.01%)⬆️
test89.51% <100.00%> (+<0.01%)⬆️
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
...Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs80.00% <100.00%> (+0.04%)⬆️
...est/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs99.09% <100.00%> (+<0.01%)⬆️
... and 3 files with indirect coverage changes
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@tarekgh
tarekgh
enabled auto-merge (squash)
February 13, 2026 00:02
@tarekgh
tarekgh
merged commit 989f5a0
into
mainFeb 13, 2026

25 of 31 checks passed


This was referenced Feb 13, 2026

Open


Open


Closed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Mar 15, 2026
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@tarekghtarekghtarekgh approved these changes

+1 more reviewer

@stephentoubstephentoubstephentoub approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@stephentoubstephentoub
Copilot code reviewCopilot

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@stephentoub@tarekgh

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add gpt-5.3 model support to TiktokenTokenizer - #7579
Merged
tarekgh merged 2 commits into
mainfrom 
copilot/add-gpt-5-3-support
Feb 13, 2026
Merged
Add gpt-5.3 model support to TiktokenTokenizer#7579
tarekgh merged 2 commits into
mainfrom 
copilot/add-gpt-5-3-support
Conversation
CopilotAI
commented
Feb 12, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Adds support for the gpt-5.3 model family to TiktokenTokenizer using O200kBase encoding.
Changes
  • TiktokenTokenizer.cs: Added gpt-5.3 and gpt-5.3- mappings to model encoding dictionaries
  • TiktokenTests.cs: Added test coverage for gpt-5.3 base and mini variants
Usage
vartokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3");vartokens=tokenizer.EncodeToIds("Hello, world!");// Also supports variantsvarminiTokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3-mini");
Follows the same pattern established for gpt-5.2 support.
Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
  • dnceng.pkgs.visualstudio.com
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet dotnet test test/Microsoft.ML.Tokenizers.Tests/Microsoft.ML.Tokenizers.Tests.csproj --filter FullyQualifiedName~TiktokenTests --no-build (dns block)
  • openaipublic.blob.core.windows.net
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.deps.json /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/testhost.dll --port 45083 --endpoint 127.0.0.1:045083 --role client --parentprocessid 6859 --telemetryoptedin false (dns block)
If you need me to access, download, or install something from one of these locations, you can either:
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.
@stephentoub
Add gpt-5.3 support in TiktokenTokenizer
cff6f1e
Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
CopilotAI
changed the title
[WIP] Add gpt-5.3 support similar to gpt-5.2Add gpt-5.3 model support to TiktokenTokenizerFeb 12, 2026
@stephentoub
stephentoub
marked this pull request as ready for review
February 12, 2026 22:13
CopilotAI
reviewed
Feb 12, 2026
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview
This pull request adds support for the GPT-5.3 model family to the TiktokenTokenizer, following the established pattern used for previous GPT model versions (gpt-5.2, gpt-5.1, etc.). The changes enable tokenization for both the base "gpt-5.3" model and its variants (e.g., "gpt-5.3-mini") using the O200kBase encoding.
Changes:
  • Added gpt-5.3 model mappings to TiktokenTokenizer for both prefix and exact matching
  • Added GPT5_3 static tokenizer property and comprehensive test coverage for the new model
  • Included test cases for both base and mini variants in the test suite
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
FileDescription
src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.csAdded "gpt-5.3-" prefix mapping and "gpt-5.3" exact name mapping to model encoding arrays, both using O200kBase encoding
test/Microsoft.ML.Tokenizers.Tests/TiktokenTests.csAdded GPT5_3 static property and test data entries for gpt-5.3 and gpt-5.3-mini variants in TestAllSupportedModelNames and TestCreationUsingModel methods
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tarekgh
tarekgh
approved these changes
Feb 12, 2026
View reviewed changes
@tarekghtarekgh
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
@codecov
codecovBot
commented
Feb 12, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
✅ All modified and coverable lines are covered by tests.

✅ Project coverage is 69.06%. Comparing base (3604580) to head (cff6f1e).
⚠️ Report is 1 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@## main #7579 +/- ##
=======================================
Coverage 69.05% 69.06% =======================================
Files 1483 1483 Lines 274362 274365 +3 Branches 28270 28270 =======================================
+ Hits 189466 189482 +16 + Misses 77510 77498 -12 + Partials 7386 7385 -1 
FlagCoverage Δ
Debug69.06% <100.00%> (+<0.01%)⬆️
production63.32% <100.00%> (+<0.01%)⬆️
test89.51% <100.00%> (+<0.01%)⬆️
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
...Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs80.00% <100.00%> (+0.04%)⬆️
...est/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs99.09% <100.00%> (+<0.01%)⬆️
... and 3 files with indirect coverage changes
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@tarekgh
tarekgh
enabled auto-merge (squash)
February 13, 2026 00:02
@tarekgh
tarekgh
merged commit 989f5a0
into
mainFeb 13, 2026

25 of 31 checks passed


This was referenced Feb 13, 2026

Open


Open


Closed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Mar 15, 2026
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@tarekghtarekghtarekgh approved these changes

+1 more reviewer

@stephentoubstephentoubstephentoub approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@stephentoubstephentoub
Copilot code reviewCopilot

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@stephentoub@tarekgh

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add gpt-5.3 model support to TiktokenTokenizer - #7579
Merged
tarekgh merged 2 commits into
mainfrom 
copilot/add-gpt-5-3-support
Feb 13, 2026
Merged
Add gpt-5.3 model support to TiktokenTokenizer#7579
tarekgh merged 2 commits into
mainfrom 
copilot/add-gpt-5-3-support
Conversation
CopilotAI
commented
Feb 12, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Adds support for the gpt-5.3 model family to TiktokenTokenizer using O200kBase encoding.
Changes
  • TiktokenTokenizer.cs: Added gpt-5.3 and gpt-5.3- mappings to model encoding dictionaries
  • TiktokenTests.cs: Added test coverage for gpt-5.3 base and mini variants
Usage
vartokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3");vartokens=tokenizer.EncodeToIds("Hello, world!");// Also supports variantsvarminiTokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3-mini");
Follows the same pattern established for gpt-5.2 support.
Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
  • dnceng.pkgs.visualstudio.com
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet dotnet test test/Microsoft.ML.Tokenizers.Tests/Microsoft.ML.Tokenizers.Tests.csproj --filter FullyQualifiedName~TiktokenTests --no-build (dns block)
  • openaipublic.blob.core.windows.net
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.deps.json /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/testhost.dll --port 45083 --endpoint 127.0.0.1:045083 --role client --parentprocessid 6859 --telemetryoptedin false (dns block)
If you need me to access, download, or install something from one of these locations, you can either:
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.
@stephentoub
Add gpt-5.3 support in TiktokenTokenizer
cff6f1e
Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
CopilotAI
changed the title
[WIP] Add gpt-5.3 support similar to gpt-5.2Add gpt-5.3 model support to TiktokenTokenizerFeb 12, 2026
@stephentoub
stephentoub
marked this pull request as ready for review
February 12, 2026 22:13
CopilotAI
reviewed
Feb 12, 2026
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview
This pull request adds support for the GPT-5.3 model family to the TiktokenTokenizer, following the established pattern used for previous GPT model versions (gpt-5.2, gpt-5.1, etc.). The changes enable tokenization for both the base "gpt-5.3" model and its variants (e.g., "gpt-5.3-mini") using the O200kBase encoding.
Changes:
  • Added gpt-5.3 model mappings to TiktokenTokenizer for both prefix and exact matching
  • Added GPT5_3 static tokenizer property and comprehensive test coverage for the new model
  • Included test cases for both base and mini variants in the test suite
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
FileDescription
src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.csAdded "gpt-5.3-" prefix mapping and "gpt-5.3" exact name mapping to model encoding arrays, both using O200kBase encoding
test/Microsoft.ML.Tokenizers.Tests/TiktokenTests.csAdded GPT5_3 static property and test data entries for gpt-5.3 and gpt-5.3-mini variants in TestAllSupportedModelNames and TestCreationUsingModel methods
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tarekgh
tarekgh
approved these changes
Feb 12, 2026
View reviewed changes
@tarekghtarekgh
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
@codecov
codecovBot
commented
Feb 12, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
✅ All modified and coverable lines are covered by tests.

✅ Project coverage is 69.06%. Comparing base (3604580) to head (cff6f1e).
⚠️ Report is 1 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@## main #7579 +/- ##
=======================================
Coverage 69.05% 69.06% =======================================
Files 1483 1483 Lines 274362 274365 +3 Branches 28270 28270 =======================================
+ Hits 189466 189482 +16 + Misses 77510 77498 -12 + Partials 7386 7385 -1 
FlagCoverage Δ
Debug69.06% <100.00%> (+<0.01%)⬆️
production63.32% <100.00%> (+<0.01%)⬆️
test89.51% <100.00%> (+<0.01%)⬆️
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
...Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs80.00% <100.00%> (+0.04%)⬆️
...est/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs99.09% <100.00%> (+<0.01%)⬆️
... and 3 files with indirect coverage changes
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@tarekgh
tarekgh
enabled auto-merge (squash)
February 13, 2026 00:02
@tarekgh
tarekgh
merged commit 989f5a0
into
mainFeb 13, 2026

25 of 31 checks passed


This was referenced Feb 13, 2026

Open


Open


Closed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Mar 15, 2026
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@tarekghtarekghtarekgh approved these changes

+1 more reviewer

@stephentoubstephentoubstephentoub approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@stephentoubstephentoub
Copilot code reviewCopilot

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@stephentoub@tarekgh

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Add gpt-5.3 model support to TiktokenTokenizer - #7579
Merged
tarekgh merged 2 commits into
mainfrom 
copilot/add-gpt-5-3-support
Feb 13, 2026
Merged
Add gpt-5.3 model support to TiktokenTokenizer#7579
tarekgh merged 2 commits into
mainfrom 
copilot/add-gpt-5-3-support
Conversation
CopilotAI
commented
Feb 12, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Adds support for the gpt-5.3 model family to TiktokenTokenizer using O200kBase encoding.
Changes
  • TiktokenTokenizer.cs: Added gpt-5.3 and gpt-5.3- mappings to model encoding dictionaries
  • TiktokenTests.cs: Added test coverage for gpt-5.3 base and mini variants
Usage
vartokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3");vartokens=tokenizer.EncodeToIds("Hello, world!");// Also supports variantsvarminiTokenizer=TiktokenTokenizer.CreateForModel("gpt-5.3-mini");
Follows the same pattern established for gpt-5.2 support.
Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
  • dnceng.pkgs.visualstudio.com
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet dotnet test test/Microsoft.ML.Tokenizers.Tests/Microsoft.ML.Tokenizers.Tests.csproj --filter FullyQualifiedName~TiktokenTests --no-build (dns block)
  • openaipublic.blob.core.windows.net
    • Triggering command: /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet /home/REDACTED/work/machinelearning/machinelearning/.dotnet/dotnet exec --runtimeconfig /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.runtimeconfig.json --depsfile /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/Microsoft.ML.Tokenizers.Tests.deps.json /home/REDACTED/work/machinelearning/machinelearning/artifacts/bin/Microsoft.ML.Tokenizers.Tests/Debug/net8.0/testhost.dll --port 45083 --endpoint 127.0.0.1:045083 --role client --parentprocessid 6859 --telemetryoptedin false (dns block)
If you need me to access, download, or install something from one of these locations, you can either:
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.
@stephentoub
Add gpt-5.3 support in TiktokenTokenizer
cff6f1e
Co-authored-by: stephentoub <2642209+stephentoub@users.noreply.github.com>
CopilotAI
changed the title
[WIP] Add gpt-5.3 support similar to gpt-5.2Add gpt-5.3 model support to TiktokenTokenizerFeb 12, 2026
@stephentoub
stephentoub
marked this pull request as ready for review
February 12, 2026 22:13
CopilotAI
reviewed
Feb 12, 2026
View reviewed changes
CopilotAI
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview
This pull request adds support for the GPT-5.3 model family to the TiktokenTokenizer, following the established pattern used for previous GPT model versions (gpt-5.2, gpt-5.1, etc.). The changes enable tokenization for both the base "gpt-5.3" model and its variants (e.g., "gpt-5.3-mini") using the O200kBase encoding.
Changes:
  • Added gpt-5.3 model mappings to TiktokenTokenizer for both prefix and exact matching
  • Added GPT5_3 static tokenizer property and comprehensive test coverage for the new model
  • Included test cases for both base and mini variants in the test suite
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
FileDescription
src/Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.csAdded "gpt-5.3-" prefix mapping and "gpt-5.3" exact name mapping to model encoding arrays, both using O200kBase encoding
test/Microsoft.ML.Tokenizers.Tests/TiktokenTests.csAdded GPT5_3 static property and test data entries for gpt-5.3 and gpt-5.3-mini variants in TestAllSupportedModelNames and TestCreationUsingModel methods
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tarekgh
tarekgh
approved these changes
Feb 12, 2026
View reviewed changes
@tarekghtarekgh
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
@codecov
codecovBot
commented
Feb 12, 2026•

edited

Loading

Copy link


Copy Markdown

Codecov Report
✅ All modified and coverable lines are covered by tests.

✅ Project coverage is 69.06%. Comparing base (3604580) to head (cff6f1e).
⚠️ Report is 1 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@## main #7579 +/- ##
=======================================
Coverage 69.05% 69.06% =======================================
Files 1483 1483 Lines 274362 274365 +3 Branches 28270 28270 =======================================
+ Hits 189466 189482 +16 + Misses 77510 77498 -12 + Partials 7386 7385 -1 
FlagCoverage Δ
Debug69.06% <100.00%> (+<0.01%)⬆️
production63.32% <100.00%> (+<0.01%)⬆️
test89.51% <100.00%> (+<0.01%)⬆️
Flags with carried forward coverage won't be shown. Click here to find out more.
Files with missing linesCoverage Δ
...Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs80.00% <100.00%> (+0.04%)⬆️
...est/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs99.09% <100.00%> (+<0.01%)⬆️
... and 3 files with indirect coverage changes
 🚀 New features to boost your workflow: 
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@tarekgh
tarekgh
enabled auto-merge (squash)
February 13, 2026 00:02
@tarekgh
tarekgh
merged commit 989f5a0
into
mainFeb 13, 2026

25 of 31 checks passed


This was referenced Feb 13, 2026

Open


Open


Closed

@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Mar 15, 2026
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@tarekghtarekghtarekgh approved these changes

+1 more reviewer

@stephentoubstephentoubstephentoub approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@stephentoubstephentoub
Copilot code reviewCopilot

Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@stephentoub@tarekgh