[LHF-04] Naive character-based token estimation logic

[sheepdestroyer]

Overview

The estimate_prompt_tokens function uses a simplistic character-count heuristic (len(content) // 4) to estimate token counts. While this provides a rough approximation for English prose, it fails significantly for code (heavy punctuation), multi-byte Unicode strings (CJK, emojis), and whitespace-padded content, compromising context-window clamping accuracy and routing metrics.

Location

File: router/main.py — estimate_prompt_tokens(), Lines 68–85

Current Code

def estimate_prompt_tokens(body: dict) -> int:
 """Estimate prompt tokens by counting characters in message contents (1 token ~= 4 chars)
 to avoid inflating metrics with large tool/schema declarations.
 """
 tokens = 0
 for msg in body.get("messages", []):
 if not isinstance(msg, dict):
 continue
 content = msg.get("content") or ""
 if isinstance(content, str):
 tokens += len(content) // 4
 elif isinstance(content, list):
 for block in content:
 if isinstance(block, dict) and block.get("type") == "text":
 tokens += len(block.get("text") or "") // 4
 tokens += 50
 return max(1, tokens)

Problems

Content Type	Actual Tokens	len//4 Estimate	Error
English prose	~100	~100	✅ ~0%
Python code	~100	~60	❌ -40% (punctuation is individually tokenized)
CJK text (Chinese/Japanese)	~50	~75	❌ +50% (multi-byte chars inflate len())
Whitespace-padded JSON	~100	~200	❌ +100% (spaces count as chars but not tokens)
System emojis	~10	~40	❌ +300% (4-byte chars)

Impact

• Severity: 🟡 Medium (Accuracy)
• Under-estimation for code prompts → context window overflows not caught by the router
• Over-estimation for CJK → unnecessarily routing to larger models
• Metrics dashboard shows inaccurate token counts, misleading cost analysis

Recommendations

Option A: Word-count baseline (Zero dependency, better accuracy)

A word-count heuristic provides tighter tracking than character division for mixed content:

def estimate_prompt_tokens(body: dict) -> int:
 """Estimate prompt tokens using word-count heuristic.
 Average English: ~1.3 tokens/word. Code: ~1.5 tokens/word.
 We use 1.3 as a conservative multiplier.
 """
 tokens = 0
 for msg in body.get("messages", []):
 if not isinstance(msg, dict):
 continue
 content = msg.get("content") or ""
 if isinstance(content, str):
 words = len(content.split())
 tokens += int(words * 1.3)
 elif isinstance(content, list):
 for block in content:
 if isinstance(block, dict) and block.get("type") == "text":
 text = block.get("text") or ""
 tokens += int(len(text.split()) * 1.3)
 tokens += 50 # metadata overhead
 return max(1, tokens)

Option B: Hybrid heuristic (Best accuracy without dependencies)

Combine character and word counting for a more robust estimate:

def estimate_prompt_tokens(body: dict) -> int:
 tokens = 0
 for msg in body.get("messages", []):
 if not isinstance(msg, dict):
 continue
 content = msg.get("content") or ""
 if isinstance(content, str):
 # Average of char-based and word-based estimates
 char_est = len(content) // 4
 word_est = int(len(content.split()) * 1.3)
 tokens += (char_est + word_est) // 2
 elif isinstance(content, list):
 for block in content:
 if isinstance(block, dict) and block.get("type") == "text":
 text = block.get("text") or ""
 char_est = len(text) // 4
 word_est = int(len(text.split()) * 1.3)
 tokens += (char_est + word_est) // 2
 tokens += 50
 return max(1, tokens)

Option C: tiktoken (Most accurate, adds dependency)

For production-grade accuracy, use OpenAI's tiktoken library:

import tiktoken
_enc = tiktoken.get_encoding("cl100k_base") # GPT-4 / Claude compatible

def estimate_prompt_tokens(body: dict) -> int:
 tokens = 0
 for msg in body.get("messages", []):
 content = msg.get("content") or ""
 if isinstance(content, str):
 tokens += len(_enc.encode(content))
 tokens += 50
 return max(1, tokens)

Note: tiktoken adds a dependency but provides exact tokenizer-aligned counts. Consider the tradeoff vs. zero-dependency heuristics.

Acceptance Criteria

• Token estimation accuracy is within ±20% for code, prose, and mixed content
• Multi-byte Unicode inputs do not produce wildly inflated estimates
• The function remains fast (sub-millisecond for typical message arrays)
• No new hard dependencies unless explicitly approved
• Existing tests pass and dashboard metrics remain consistent

[google-labs-jules]

Jules is on it. When finished, you will see another comment and be able to review a PR.

[google-labs-jules]

Ready for a review! A PR has been created.