TOON Format for Python

⚠️ Early Release (v0.1.x): The API is stable enough for use, but may evolve before 1.0.0.

Compact, human-readable serialization format for LLM contexts with strong token savings on mixed and nested data. Uses a minimal syntax and deterministic round-trips. Designed for fast encode/decode and practical prompt efficiency.

Key Features

Minimal syntax • Tabular arrays for uniform data • Optional auto format selection • Python 3.9+

Installation

pip install p-toon-llm

Quick Start

fromtoon_formatimportencode, decode# Simple objectencode({"name": "Alice", "age": 30})
# {name,age|Alice|30}# Tabular array (uniform objects)encode([{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}])
# ^csv[id,name|1,Alice|2,Bob]# Decode back to Pythondecode("{items|[apple|banana]}")
# {'items': ['apple', 'banana']}

API Reference

encode(value, options=None) → str

encode({"id": 123})
encode({"id": 123}, {"mode": "auto", "candidates": ("toon", "json", "csv")})

Options:

1. mode: toon (default), hybrid, or auto
2. candidates: iterable of formats for auto mode
3. metric: tokens or chars for auto mode

decode(input_str, options=None) → Any

decode("{id|123}")

Token Counting & Comparison

fromtoon_formatimportestimate_savings, compare_formats, count_tokensdata= {"users": [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]}
result=estimate_savings(data)
print(f"Saves {result['savings_percent']:.1f}% tokens")
print(compare_formats(data))
toon_str=encode(data)
print(count_tokens(toon_str))

Requires tiktoken for accurate token counts. Without it, count_tokens falls back to character length.

Format Specification

Type	Example Input	TOON Output
Object	{"name": "Alice", "age": 30}	`{name,age
Primitive Array	[1, 2, 3]	`[1
Tabular Array	[{"id": 1, "name": "A"}, {"id": 2, "name": "B"}]	`^csv[id,name
Mixed Array	[{"x": 1}, 42, "hi"]	`[{x

Quoting: only when necessary (empty, reserved tokens, numeric ambiguity, whitespace, delimiters)

Type Normalization: datetime/date -> ISO 8601 • Decimal -> float • NaN/Inf -> null • -0 -> 0

Auto Mode

fromtoon_formatimportencode_best, encodebest=encode_best(data, candidates=("toon", "json", "yaml", "csv"))
print(best["format"], best["text"])
print(encode(data, options={"mode": "auto", "candidates": ("toon", "json", "csv")}))

Benchmarks

TOON tends to win on mixed and nested data. CSV will usually win on flat tables. Use your own datasets for reliable results.

fromtoon_formatimportcompare_formats, estimate_savingsprint(compare_formats(data))
print(estimate_savings(data))

Example dataset:

data= {
"users": [
{"id": 1, "name": "Alice", "role": "admin"},
{"id": 2, "name": "Bob", "role": "member"},
{"id": 3, "name": "Cara", "role": "member"},
],
"meta": {"team": "Toon", "active": True, "region": "EU"},
"events": [
{"type": "login", "user_id": 1, "ok": True},
{"type": "purchase", "user_id": 2, "amount": 19.99},
"note: rollout 1",
],
}

Example results (tokens and chars will vary by tokenizer and data):

Format	Tokens	Size (chars)	Notes
TOON	203	203	Compact mixed/nested
JSON compact	293	293	Minified JSON
JSON pretty	508	508	Indented JSON
YAML	322	322	Simple YAML
CSV (users)	53	53	Table only

Development

python -m pytest

Documentation

1. docs/index.md
2. docs/format.md
3. docs/api.md
4. docs/performance.md
5. docs/prompt_header.md

License

MIT License – see LICENSE