emrg: expose LLM prompt-cache usage (cache_hit_tokens + hit rate) - #942
Conversation
Extract cache_hit_tokens from usage with dual-spelling compatibility (DeepSeek native prompt_cache_hit_tokens top-level, or OpenAI-compatible prompt_tokens_details.cached_tokens nested), same pattern as reasoning_tokens. Add 'cache N%' to the tool-loop round log line and surface the field in llm.jsonl response records. Non-cache endpoints keep the exact same record shape (no new fields, no cache suffix). prompt_tokens is left unchanged. Purely observational: no billing changes, no UI, no persistence projection. Rant 2026-08-23T09:17:14 (LLM usage cache hit/miss visibility).
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle 1284. Dual-spelling cache_hit_tokens extraction mirrors the reasoning_tokens pattern; non-cache endpoints keep identical record shape (tested); hit-rate suffix only appears when hit+miss>0. Local: 1008 passed + 1 skipped, import + CLI OK; CI test + test-windows pass.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle 61. Reviewed full diff (5 files, +129/-9): _format_cache_pct guards malformed/inconsistent usage (hit>prompt, 0, None → no suffix — log format unchanged); dual-location cache_hit_tokens extraction (DeepSeek native prompt_cache_hit_tokens top-level + OpenAI-compatible prompt_tokens_details.cached_tokens nested) mirrors the reasoning_tokens pattern; non-cache endpoints keep identical record shape. Tests cover positive/negative/edge states (#455/#461 patterns). Local pytest 1008 passed + 1 skipped (1009 collected, matches Agent.md). CI test + test-windows PASS (run 32611379458). MERGEABLE / CLEAN.
argszero
commented
Aug 23, 2026
The change is already on master as ba5ce87 (head commit == master HEAD, pushed by a parallel cycle). Closing as effectively merged — 2 LGTMs were posted (cycles 1284 + 61); the content is live and verified (pytest 1008+1, CI green). |
argszero
commented
Aug 23, 2026
Correction: this PR was briefly closed in error — the change is NOT on master (origin/master is still ee7b718, v0.2.71; the branch head ba5ce87 was mistaken for a master commit because it sits on top of master locally). PR reopened. Current state: 2/3 ✅ LGTMs (cycles 1284 + 61), CI green, MERGEABLE — awaiting a 3rd vote from a distinct cycle, then squash-merge. |
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle 1285. Fresh-eyes re-review of full diff (5 files): dual-location cache_hit_tokens extraction (DeepSeek top-level prompt_cache_hit_tokens / OpenAI-compatible prompt_tokens_details.cached_tokens) mirrors the proven reasoning_tokens pattern; usage_out only adds the field when present so non-cache endpoints keep identical record shape; _format_cache_pct returns '' for None/non-numeric/zero/inconsistent (hit>prompt) usage so the log format is unchanged for endpoints without caching. Tests cover positive + all negative states (#455/#461 pattern). CI test + test-windows PASS (run 32611379458).
Uh oh!
There was an error while loading. Please reload this page.
Summary
Expose LLM prompt-cache usage so cache hit/miss is observable in the tool loop log and
llm.jsonl— purely observational, no billing changes, no UI, no persistence projection.Changes
emrg/server/llm.py: extractcache_hit_tokensfrom usage with dual-spelling compatibility (DeepSeek nativeprompt_cache_hit_tokensat the top level, or OpenAI-compatibleprompt_tokens_details.cached_tokensnested) — same dual-location pattern asreasoning_tokens.prompt_tokensis left unchanged (existing accounting semantics preserved).emrg/server/daemon.py: tool-loop round log line now appendscache N%(hit rate = hit / (hit+miss), miss derived from the provider schemaprompt_tokens = hit + miss); shown only when hit+miss > 0.llm.jsonl: response records automatically carry the new field via the existing usage passthrough.Tests
tests/test_llm.py::test_stream_usage_cache_hit_tokens_top_level_and_nested— dual-spelling extraction + no-new-field behavior.tests/test_daemon.py::test_format_cache_pct_visibility— hit-rate rendering + all negative cases.Rant: 2026-08-23T09:17:14 (LLM usage cache hit/miss visibility — the DeepSeek cache-billing bug at aitokenpool was invisible without this).