feat(cli): add --stats flag for token comparison - #41

Open
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats
Open

feat(cli): add --stats flag for token comparison#41
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats

Conversation

@therrshan

@therrshantherrshan commented Nov 16, 2025

Copy link
Copy Markdown

Description

Adds --stats flag to the CLI for displaying token count statistics when encoding JSON to TOON. Uses the existing compare_formats() utility from utils.py to show a comparison table with token counts and savings percentage.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test coverage improvement

Related Issues

Closes #

Changes Made

  • Added --stats argument to CLI argparse configuration
  • Integrated existing compare_formats() function from utils.py
  • Added error handling for missing tiktoken with helpful installation message
  • Added comprehensive test coverage in test_cli.py
  • Updated README.md to document the new --stats flag

SPEC Compliance

  • This PR implements/fixes spec compliance
  • Spec section(s) affected:
  • Spec version:

Testing

  • All existing tests pass
  • Added new tests for changes
  • Tested on Python 3.8
  • Tested on Python 3.9
  • Tested on Python 3.10
  • Tested on Python 3.11
  • Tested on Python 3.12

Test Output

799 passed, 13 skipped in 3.85s
TOTAL 1123 76 93.23%
Coverage : 93.23%

Code Quality

  • Ran ruff check src/toon_format tests - no issues
  • Ran ruff format src/toon_format tests - code formatted
  • Ran mypy src/toon_format - no critical errors
  • All tests pass: pytest tests/ -v

Checklist

  • My code follows the project's coding standards (PEP 8, line length 100)
  • I have added type hints to new code
  • I have added tests that prove my fix/feature works
  • New and existing tests pass locally
  • I have updated documentation (README.md, CLAUDE.md if needed)
  • My changes do not introduce new dependencies
  • I have maintained Python 3.8+ compatibility
  • I have reviewed the TOON specification for relevant sections

Performance Impact

  • No performance impact
  • Performance improvement (describe below)
  • Potential performance regression (describe and justify below)

Breaking Changes

  • No breaking changes
  • Breaking changes (describe migration path below)

Screenshots / Examples

$ echo'{"users": [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]}'| uv run toon - --stats

Output:

Format Comparison
────────────────────────────────────────────────
Format Tokens Size (chars)
JSON 45 117
TOON 19 36
────────────────────────────────────────────────
Savings: 26 tokens (57.8%)
users[2]{id,name}:
1,Alice
2,Bob

Additional Context

This is my first contribution to toon-python. The feature leverages existing token counting utilities from utils.py that were already part of the public API but not exposed in the CLI. No new dependencies required - uses existing tiktoken from the benchmark dependency group.

Checklist for Reviewers

  • Code changes are clear and well-documented
  • Tests adequately cover the changes
  • Documentation is updated
  • No security concerns
  • Follows TOON specification
  • Backward compatible (or breaking changes are justified and documented)

@johannschopplichjohannschopplich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (README-wise only).

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@therrshan tests are broken, can you please look into it? Thanks

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@therrshan@alesanfra@johannschopplich
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

feat(cli): add --stats flag for token comparison - #41

Open
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats
Open

feat(cli): add --stats flag for token comparison#41
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats

Conversation

@therrshan

@therrshantherrshan commented Nov 16, 2025

Copy link
Copy Markdown

Description

Adds --stats flag to the CLI for displaying token count statistics when encoding JSON to TOON. Uses the existing compare_formats() utility from utils.py to show a comparison table with token counts and savings percentage.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test coverage improvement

Related Issues

Closes #

Changes Made

  • Added --stats argument to CLI argparse configuration
  • Integrated existing compare_formats() function from utils.py
  • Added error handling for missing tiktoken with helpful installation message
  • Added comprehensive test coverage in test_cli.py
  • Updated README.md to document the new --stats flag

SPEC Compliance

  • This PR implements/fixes spec compliance
  • Spec section(s) affected:
  • Spec version:

Testing

  • All existing tests pass
  • Added new tests for changes
  • Tested on Python 3.8
  • Tested on Python 3.9
  • Tested on Python 3.10
  • Tested on Python 3.11
  • Tested on Python 3.12

Test Output

799 passed, 13 skipped in 3.85s
TOTAL 1123 76 93.23%
Coverage : 93.23%

Code Quality

  • Ran ruff check src/toon_format tests - no issues
  • Ran ruff format src/toon_format tests - code formatted
  • Ran mypy src/toon_format - no critical errors
  • All tests pass: pytest tests/ -v

Checklist

  • My code follows the project's coding standards (PEP 8, line length 100)
  • I have added type hints to new code
  • I have added tests that prove my fix/feature works
  • New and existing tests pass locally
  • I have updated documentation (README.md, CLAUDE.md if needed)
  • My changes do not introduce new dependencies
  • I have maintained Python 3.8+ compatibility
  • I have reviewed the TOON specification for relevant sections

Performance Impact

  • No performance impact
  • Performance improvement (describe below)
  • Potential performance regression (describe and justify below)

Breaking Changes

  • No breaking changes
  • Breaking changes (describe migration path below)

Screenshots / Examples

$ echo'{"users": [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]}'| uv run toon - --stats

Output:

Format Comparison
────────────────────────────────────────────────
Format Tokens Size (chars)
JSON 45 117
TOON 19 36
────────────────────────────────────────────────
Savings: 26 tokens (57.8%)
users[2]{id,name}:
1,Alice
2,Bob

Additional Context

This is my first contribution to toon-python. The feature leverages existing token counting utilities from utils.py that were already part of the public API but not exposed in the CLI. No new dependencies required - uses existing tiktoken from the benchmark dependency group.

Checklist for Reviewers

  • Code changes are clear and well-documented
  • Tests adequately cover the changes
  • Documentation is updated
  • No security concerns
  • Follows TOON specification
  • Backward compatible (or breaking changes are justified and documented)

@johannschopplichjohannschopplich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (README-wise only).

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@therrshan tests are broken, can you please look into it? Thanks

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@therrshan@alesanfra@johannschopplich
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(cli): add --stats flag for token comparison - #41

Open
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats
Open

feat(cli): add --stats flag for token comparison#41
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats

Conversation

@therrshan

@therrshantherrshan commented Nov 16, 2025

Copy link
Copy Markdown

Description

Adds --stats flag to the CLI for displaying token count statistics when encoding JSON to TOON. Uses the existing compare_formats() utility from utils.py to show a comparison table with token counts and savings percentage.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test coverage improvement

Related Issues

Closes #

Changes Made

  • Added --stats argument to CLI argparse configuration
  • Integrated existing compare_formats() function from utils.py
  • Added error handling for missing tiktoken with helpful installation message
  • Added comprehensive test coverage in test_cli.py
  • Updated README.md to document the new --stats flag

SPEC Compliance

  • This PR implements/fixes spec compliance
  • Spec section(s) affected:
  • Spec version:

Testing

  • All existing tests pass
  • Added new tests for changes
  • Tested on Python 3.8
  • Tested on Python 3.9
  • Tested on Python 3.10
  • Tested on Python 3.11
  • Tested on Python 3.12

Test Output

799 passed, 13 skipped in 3.85s
TOTAL 1123 76 93.23%
Coverage : 93.23%

Code Quality

  • Ran ruff check src/toon_format tests - no issues
  • Ran ruff format src/toon_format tests - code formatted
  • Ran mypy src/toon_format - no critical errors
  • All tests pass: pytest tests/ -v

Checklist

  • My code follows the project's coding standards (PEP 8, line length 100)
  • I have added type hints to new code
  • I have added tests that prove my fix/feature works
  • New and existing tests pass locally
  • I have updated documentation (README.md, CLAUDE.md if needed)
  • My changes do not introduce new dependencies
  • I have maintained Python 3.8+ compatibility
  • I have reviewed the TOON specification for relevant sections

Performance Impact

  • No performance impact
  • Performance improvement (describe below)
  • Potential performance regression (describe and justify below)

Breaking Changes

  • No breaking changes
  • Breaking changes (describe migration path below)

Screenshots / Examples

$ echo'{"users": [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]}'| uv run toon - --stats

Output:

Format Comparison
────────────────────────────────────────────────
Format Tokens Size (chars)
JSON 45 117
TOON 19 36
────────────────────────────────────────────────
Savings: 26 tokens (57.8%)
users[2]{id,name}:
1,Alice
2,Bob

Additional Context

This is my first contribution to toon-python. The feature leverages existing token counting utilities from utils.py that were already part of the public API but not exposed in the CLI. No new dependencies required - uses existing tiktoken from the benchmark dependency group.

Checklist for Reviewers

  • Code changes are clear and well-documented
  • Tests adequately cover the changes
  • Documentation is updated
  • No security concerns
  • Follows TOON specification
  • Backward compatible (or breaking changes are justified and documented)

@johannschopplichjohannschopplich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (README-wise only).

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@therrshan tests are broken, can you please look into it? Thanks

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@therrshan@alesanfra@johannschopplich
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(cli): add --stats flag for token comparison - #41

Open
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats
Open

feat(cli): add --stats flag for token comparison#41
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats

Conversation

@therrshan

@therrshantherrshan commented Nov 16, 2025

Copy link
Copy Markdown

Description

Adds --stats flag to the CLI for displaying token count statistics when encoding JSON to TOON. Uses the existing compare_formats() utility from utils.py to show a comparison table with token counts and savings percentage.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test coverage improvement

Related Issues

Closes #

Changes Made

  • Added --stats argument to CLI argparse configuration
  • Integrated existing compare_formats() function from utils.py
  • Added error handling for missing tiktoken with helpful installation message
  • Added comprehensive test coverage in test_cli.py
  • Updated README.md to document the new --stats flag

SPEC Compliance

  • This PR implements/fixes spec compliance
  • Spec section(s) affected:
  • Spec version:

Testing

  • All existing tests pass
  • Added new tests for changes
  • Tested on Python 3.8
  • Tested on Python 3.9
  • Tested on Python 3.10
  • Tested on Python 3.11
  • Tested on Python 3.12

Test Output

799 passed, 13 skipped in 3.85s
TOTAL 1123 76 93.23%
Coverage : 93.23%

Code Quality

  • Ran ruff check src/toon_format tests - no issues
  • Ran ruff format src/toon_format tests - code formatted
  • Ran mypy src/toon_format - no critical errors
  • All tests pass: pytest tests/ -v

Checklist

  • My code follows the project's coding standards (PEP 8, line length 100)
  • I have added type hints to new code
  • I have added tests that prove my fix/feature works
  • New and existing tests pass locally
  • I have updated documentation (README.md, CLAUDE.md if needed)
  • My changes do not introduce new dependencies
  • I have maintained Python 3.8+ compatibility
  • I have reviewed the TOON specification for relevant sections

Performance Impact

  • No performance impact
  • Performance improvement (describe below)
  • Potential performance regression (describe and justify below)

Breaking Changes

  • No breaking changes
  • Breaking changes (describe migration path below)

Screenshots / Examples

$ echo'{"users": [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]}'| uv run toon - --stats

Output:

Format Comparison
────────────────────────────────────────────────
Format Tokens Size (chars)
JSON 45 117
TOON 19 36
────────────────────────────────────────────────
Savings: 26 tokens (57.8%)
users[2]{id,name}:
1,Alice
2,Bob

Additional Context

This is my first contribution to toon-python. The feature leverages existing token counting utilities from utils.py that were already part of the public API but not exposed in the CLI. No new dependencies required - uses existing tiktoken from the benchmark dependency group.

Checklist for Reviewers

  • Code changes are clear and well-documented
  • Tests adequately cover the changes
  • Documentation is updated
  • No security concerns
  • Follows TOON specification
  • Backward compatible (or breaking changes are justified and documented)

@johannschopplichjohannschopplich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (README-wise only).

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@therrshan tests are broken, can you please look into it? Thanks

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@therrshan@alesanfra@johannschopplich
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

feat(cli): add --stats flag for token comparison - #41

Open
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats
Open

feat(cli): add --stats flag for token comparison#41
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats

Conversation

@therrshan

@therrshantherrshan commented Nov 16, 2025

Copy link
Copy Markdown

Description

Adds --stats flag to the CLI for displaying token count statistics when encoding JSON to TOON. Uses the existing compare_formats() utility from utils.py to show a comparison table with token counts and savings percentage.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test coverage improvement

Related Issues

Closes #

Changes Made

  • Added --stats argument to CLI argparse configuration
  • Integrated existing compare_formats() function from utils.py
  • Added error handling for missing tiktoken with helpful installation message
  • Added comprehensive test coverage in test_cli.py
  • Updated README.md to document the new --stats flag

SPEC Compliance

  • This PR implements/fixes spec compliance
  • Spec section(s) affected:
  • Spec version:

Testing

  • All existing tests pass
  • Added new tests for changes
  • Tested on Python 3.8
  • Tested on Python 3.9
  • Tested on Python 3.10
  • Tested on Python 3.11
  • Tested on Python 3.12

Test Output

799 passed, 13 skipped in 3.85s
TOTAL 1123 76 93.23%
Coverage : 93.23%

Code Quality

  • Ran ruff check src/toon_format tests - no issues
  • Ran ruff format src/toon_format tests - code formatted
  • Ran mypy src/toon_format - no critical errors
  • All tests pass: pytest tests/ -v

Checklist

  • My code follows the project's coding standards (PEP 8, line length 100)
  • I have added type hints to new code
  • I have added tests that prove my fix/feature works
  • New and existing tests pass locally
  • I have updated documentation (README.md, CLAUDE.md if needed)
  • My changes do not introduce new dependencies
  • I have maintained Python 3.8+ compatibility
  • I have reviewed the TOON specification for relevant sections

Performance Impact

  • No performance impact
  • Performance improvement (describe below)
  • Potential performance regression (describe and justify below)

Breaking Changes

  • No breaking changes
  • Breaking changes (describe migration path below)

Screenshots / Examples

$ echo'{"users": [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]}'| uv run toon - --stats

Output:

Format Comparison
────────────────────────────────────────────────
Format Tokens Size (chars)
JSON 45 117
TOON 19 36
────────────────────────────────────────────────
Savings: 26 tokens (57.8%)
users[2]{id,name}:
1,Alice
2,Bob

Additional Context

This is my first contribution to toon-python. The feature leverages existing token counting utilities from utils.py that were already part of the public API but not exposed in the CLI. No new dependencies required - uses existing tiktoken from the benchmark dependency group.

Checklist for Reviewers

  • Code changes are clear and well-documented
  • Tests adequately cover the changes
  • Documentation is updated
  • No security concerns
  • Follows TOON specification
  • Backward compatible (or breaking changes are justified and documented)

@johannschopplichjohannschopplich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (README-wise only).

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@therrshan tests are broken, can you please look into it? Thanks

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@therrshan@alesanfra@johannschopplich
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(cli): add --stats flag for token comparison - #41

Open
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats
Open

feat(cli): add --stats flag for token comparison#41
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats

Conversation

@therrshan

@therrshantherrshan commented Nov 16, 2025

Copy link
Copy Markdown

Description

Adds --stats flag to the CLI for displaying token count statistics when encoding JSON to TOON. Uses the existing compare_formats() utility from utils.py to show a comparison table with token counts and savings percentage.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test coverage improvement

Related Issues

Closes #

Changes Made

  • Added --stats argument to CLI argparse configuration
  • Integrated existing compare_formats() function from utils.py
  • Added error handling for missing tiktoken with helpful installation message
  • Added comprehensive test coverage in test_cli.py
  • Updated README.md to document the new --stats flag

SPEC Compliance

  • This PR implements/fixes spec compliance
  • Spec section(s) affected:
  • Spec version:

Testing

  • All existing tests pass
  • Added new tests for changes
  • Tested on Python 3.8
  • Tested on Python 3.9
  • Tested on Python 3.10
  • Tested on Python 3.11
  • Tested on Python 3.12

Test Output

799 passed, 13 skipped in 3.85s
TOTAL 1123 76 93.23%
Coverage : 93.23%

Code Quality

  • Ran ruff check src/toon_format tests - no issues
  • Ran ruff format src/toon_format tests - code formatted
  • Ran mypy src/toon_format - no critical errors
  • All tests pass: pytest tests/ -v

Checklist

  • My code follows the project's coding standards (PEP 8, line length 100)
  • I have added type hints to new code
  • I have added tests that prove my fix/feature works
  • New and existing tests pass locally
  • I have updated documentation (README.md, CLAUDE.md if needed)
  • My changes do not introduce new dependencies
  • I have maintained Python 3.8+ compatibility
  • I have reviewed the TOON specification for relevant sections

Performance Impact

  • No performance impact
  • Performance improvement (describe below)
  • Potential performance regression (describe and justify below)

Breaking Changes

  • No breaking changes
  • Breaking changes (describe migration path below)

Screenshots / Examples

$ echo'{"users": [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]}'| uv run toon - --stats

Output:

Format Comparison
────────────────────────────────────────────────
Format Tokens Size (chars)
JSON 45 117
TOON 19 36
────────────────────────────────────────────────
Savings: 26 tokens (57.8%)
users[2]{id,name}:
1,Alice
2,Bob

Additional Context

This is my first contribution to toon-python. The feature leverages existing token counting utilities from utils.py that were already part of the public API but not exposed in the CLI. No new dependencies required - uses existing tiktoken from the benchmark dependency group.

Checklist for Reviewers

  • Code changes are clear and well-documented
  • Tests adequately cover the changes
  • Documentation is updated
  • No security concerns
  • Follows TOON specification
  • Backward compatible (or breaking changes are justified and documented)

@johannschopplichjohannschopplich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (README-wise only).

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@therrshan tests are broken, can you please look into it? Thanks

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@therrshan@alesanfra@johannschopplich
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(cli): add --stats flag for token comparison - #41

Open
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats
Open

feat(cli): add --stats flag for token comparison#41
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats

Conversation

@therrshan

@therrshantherrshan commented Nov 16, 2025

Copy link
Copy Markdown

Description

Adds --stats flag to the CLI for displaying token count statistics when encoding JSON to TOON. Uses the existing compare_formats() utility from utils.py to show a comparison table with token counts and savings percentage.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test coverage improvement

Related Issues

Closes #

Changes Made

  • Added --stats argument to CLI argparse configuration
  • Integrated existing compare_formats() function from utils.py
  • Added error handling for missing tiktoken with helpful installation message
  • Added comprehensive test coverage in test_cli.py
  • Updated README.md to document the new --stats flag

SPEC Compliance

  • This PR implements/fixes spec compliance
  • Spec section(s) affected:
  • Spec version:

Testing

  • All existing tests pass
  • Added new tests for changes
  • Tested on Python 3.8
  • Tested on Python 3.9
  • Tested on Python 3.10
  • Tested on Python 3.11
  • Tested on Python 3.12

Test Output

799 passed, 13 skipped in 3.85s
TOTAL 1123 76 93.23%
Coverage : 93.23%

Code Quality

  • Ran ruff check src/toon_format tests - no issues
  • Ran ruff format src/toon_format tests - code formatted
  • Ran mypy src/toon_format - no critical errors
  • All tests pass: pytest tests/ -v

Checklist

  • My code follows the project's coding standards (PEP 8, line length 100)
  • I have added type hints to new code
  • I have added tests that prove my fix/feature works
  • New and existing tests pass locally
  • I have updated documentation (README.md, CLAUDE.md if needed)
  • My changes do not introduce new dependencies
  • I have maintained Python 3.8+ compatibility
  • I have reviewed the TOON specification for relevant sections

Performance Impact

  • No performance impact
  • Performance improvement (describe below)
  • Potential performance regression (describe and justify below)

Breaking Changes

  • No breaking changes
  • Breaking changes (describe migration path below)

Screenshots / Examples

$ echo'{"users": [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]}'| uv run toon - --stats

Output:

Format Comparison
────────────────────────────────────────────────
Format Tokens Size (chars)
JSON 45 117
TOON 19 36
────────────────────────────────────────────────
Savings: 26 tokens (57.8%)
users[2]{id,name}:
1,Alice
2,Bob

Additional Context

This is my first contribution to toon-python. The feature leverages existing token counting utilities from utils.py that were already part of the public API but not exposed in the CLI. No new dependencies required - uses existing tiktoken from the benchmark dependency group.

Checklist for Reviewers

  • Code changes are clear and well-documented
  • Tests adequately cover the changes
  • Documentation is updated
  • No security concerns
  • Follows TOON specification
  • Backward compatible (or breaking changes are justified and documented)

@johannschopplichjohannschopplich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (README-wise only).

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@therrshan tests are broken, can you please look into it? Thanks

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@therrshan@alesanfra@johannschopplich
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

feat(cli): add --stats flag for token comparison - #41

Open
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats
Open

feat(cli): add --stats flag for token comparison#41
therrshan wants to merge 2 commits into
toon-format:mainfrom
therrshan:feat/add-cli-stats

Conversation

@therrshan

@therrshantherrshan commented Nov 16, 2025

Copy link
Copy Markdown

Description

Adds --stats flag to the CLI for displaying token count statistics when encoding JSON to TOON. Uses the existing compare_formats() utility from utils.py to show a comparison table with token counts and savings percentage.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test coverage improvement

Related Issues

Closes #

Changes Made

  • Added --stats argument to CLI argparse configuration
  • Integrated existing compare_formats() function from utils.py
  • Added error handling for missing tiktoken with helpful installation message
  • Added comprehensive test coverage in test_cli.py
  • Updated README.md to document the new --stats flag

SPEC Compliance

  • This PR implements/fixes spec compliance
  • Spec section(s) affected:
  • Spec version:

Testing

  • All existing tests pass
  • Added new tests for changes
  • Tested on Python 3.8
  • Tested on Python 3.9
  • Tested on Python 3.10
  • Tested on Python 3.11
  • Tested on Python 3.12

Test Output

799 passed, 13 skipped in 3.85s
TOTAL 1123 76 93.23%
Coverage : 93.23%

Code Quality

  • Ran ruff check src/toon_format tests - no issues
  • Ran ruff format src/toon_format tests - code formatted
  • Ran mypy src/toon_format - no critical errors
  • All tests pass: pytest tests/ -v

Checklist

  • My code follows the project's coding standards (PEP 8, line length 100)
  • I have added type hints to new code
  • I have added tests that prove my fix/feature works
  • New and existing tests pass locally
  • I have updated documentation (README.md, CLAUDE.md if needed)
  • My changes do not introduce new dependencies
  • I have maintained Python 3.8+ compatibility
  • I have reviewed the TOON specification for relevant sections

Performance Impact

  • No performance impact
  • Performance improvement (describe below)
  • Potential performance regression (describe and justify below)

Breaking Changes

  • No breaking changes
  • Breaking changes (describe migration path below)

Screenshots / Examples

$ echo'{"users": [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]}'| uv run toon - --stats

Output:

Format Comparison
────────────────────────────────────────────────
Format Tokens Size (chars)
JSON 45 117
TOON 19 36
────────────────────────────────────────────────
Savings: 26 tokens (57.8%)
users[2]{id,name}:
1,Alice
2,Bob

Additional Context

This is my first contribution to toon-python. The feature leverages existing token counting utilities from utils.py that were already part of the public API but not exposed in the CLI. No new dependencies required - uses existing tiktoken from the benchmark dependency group.

Checklist for Reviewers

  • Code changes are clear and well-documented
  • Tests adequately cover the changes
  • Documentation is updated
  • No security concerns
  • Follows TOON specification
  • Backward compatible (or breaking changes are justified and documented)

@johannschopplichjohannschopplich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (README-wise only).

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alesanfraalesanfra left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@therrshan tests are broken, can you please look into it? Thanks

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@therrshan@alesanfra@johannschopplich