TOON Format for Python

⚠️ Beta Status (v0.9.x): This library is in active development and working towards spec compliance. Beta published to PyPI. API may change before 1.0.0 release.

Compact, human-readable serialization format for LLM contexts with 30-60% token reduction vs JSON. Combines YAML-like indentation with CSV-like tabular arrays. Working towards full compatibility with the official TOON specification.

Key Features: Minimal syntax • Tabular arrays for uniform data • Array length validation • Python 3.8+ • Comprehensive test coverage.

# Beta published to PyPI - install from source:
git clone https://github.com/toon-format/toon-python.git
cd toon-python
uv sync
# Or install directly from GitHub:
pip install git+https://github.com/toon-format/toon-python.git

Quick Start

fromtoon_formatimportencode, decode# Simple objectencode({"name": "Alice", "age": 30})
# name: Alice# age: 30# Tabular array (uniform objects)encode([{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}])
# [2,]{id,name}:# 1,Alice# 2,Bob# Decode back to Pythondecode("items[2]: apple,banana")
# {'items': ['apple', 'banana']}

CLI Usage

# Auto-detect format by extension
toon input.json -o output.toon # Encode
toon data.toon -o output.json # Decodeecho'{"x": 1}'| toon - # Stdin/stdout# Options
toon data.json --encode --delimiter "\t" --length-marker
toon data.toon --decode --no-strict --indent 4

Options:-e/--encode-d/--decode-o/--output--delimiter--indent--length-marker--no-strict

API Reference

encode(value, options=None) → str

encode({"id": 123}, {"delimiter": "\t", "indent": 4, "lengthMarker": "#"})

Options:

• delimiter: "," (default), "\t", "|"
• indent: Spaces per level (default: 2)
• lengthMarker: "" (default) or "#" to prefix array lengths

decode(input_str, options=None) → Any

decode("id: 123", {"indent": 2, "strict": True})

Options:

• indent: Expected indent size (default: 2)
• strict: Validate syntax, lengths, delimiters (default: True)

Token Counting & Comparison

Measure token efficiency and compare formats:

fromtoon_formatimportestimate_savings, compare_formats, count_tokens# Measure savingsdata= {"users": [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]}
result=estimate_savings(data)
print(f"Saves {result['savings_percent']:.1f}% tokens") # Saves 42.3% tokens# Visual comparisonprint(compare_formats(data))
# Format Comparison# ────────────────────────────────────────────────# Format Tokens Size (chars)# JSON 45 123# TOON 28 85# ────────────────────────────────────────────────# Savings: 17 tokens (37.8%)# Count tokens directlytoon_str=encode(data)
tokens=count_tokens(toon_str) # Uses tiktoken (gpt5/gpt5-mini)

Requires tiktoken:uv add tiktoken (benchmark features are optional)

Format Specification

Type	Example Input	TOON Output
Object	{"name": "Alice", "age": 30}	name: Alice age: 30
Primitive Array	[1, 2, 3]	[3]: 1,2,3
Tabular Array	[{"id": 1, "name": "A"}, {"id": 2, "name": "B"}]	[2,]{id,name}: 1,A 2,B
Mixed Array	[{"x": 1}, 42, "hi"]	[3]: - x: 1 - 42 - hi

Quoting: Only when necessary (empty, keywords, numeric strings, whitespace, structural chars, delimiters)

Type Normalization:Infinity/NaN/Functions → null • Decimal → float • datetime → ISO 8601 • -0 → 0

Pydantic Integration – (Structured TOON for LLM Outputs)

Adds a completely optional Pydantic integration via the [pydantic] extra.

pip install "toon-python[pydantic]"

Features

• Schema: 50–60 % smaller than model_json_schema()
• Zero JSON parsing errors
• Works with Instructor, Outlines, Marvin, LangChain agents, etc.
• Full Pydantic validation preserved

Usage After Release

fromtoon_format.pydanticimportToonPydanticModelclassUser(ToonPydanticModel):
name: strage: intemail: str|None=None# Convert schema to TOON for LLM system promptsschema_toon=User.schema_to_toon()
# name:str,age:int,email:str|None# Parse LLM TOON output into validated Pydantic modeltoon_output="name:Ansar,age:25,email:ansar@example.com"user=User.model_validate_toon(toon_output)
# user.name → "Ansar"# user.age → 25# user.email → "ansar@example.com"# Serialize a model instance back to TOONtoon_str=user.model_dump_toon()

Development

# Setup (requires uv: https://docs.astral.sh/uv/)
git clone https://github.com/toon-format/toon-python.git
cd toon-python
uv sync
# Run tests (792 tests, 91% coverage, 85% enforced)
uv run pytest --cov=toon_format --cov-report=term
# Code quality
uv run ruff check src/ tests/ # Lint
uv run ruff format src/ tests/ # Format
uv run mypy src/ # Type check

CI/CD: GitHub Actions • Python 3.8-3.14 • Coverage enforcement • PR coverage comments

Project Status & Roadmap

Following semantic versioning towards 1.0.0:

• v0.8.x - Initial code set, tests, documentation ✅
• v0.9.x - Serializer improvements, spec compliance testing, publishing setup (current)
• v1.0.0-rc.x - Release candidates for production readiness
• v1.0.0 - First stable release with full spec compliance

See CONTRIBUTING.md for detailed guidelines.

Documentation

• 📘 Full Documentation - Complete guides and references
• 🔧 API Reference - Detailed function documentation
• 📋 Format Specification - TOON syntax and rules
• 🤖 LLM Integration - Best practices for LLM usage
• 📜 TOON Spec - Official specification
• 🐛 Issues - Bug reports and features
• 🤝 Contributing - Contribution guidelines

Contributors

• Xavi Vinaixa
• David Pirogov
• Justar
• Johann Schopplich

License

MIT License – see LICENSE for details