Skip to content

emrg: expose LLM prompt-cache usage (cache_hit_tokens + hit rate) - #942

Merged
argszero merged 1 commit into
masterfrom
feature/llm-usage-cache-visibility
Aug 23, 2026
Merged

emrg: expose LLM prompt-cache usage (cache_hit_tokens + hit rate)#942
argszero merged 1 commit into
masterfrom
feature/llm-usage-cache-visibility

Conversation

@argszero

Copy link
Copy Markdown
Owner

Summary

Expose LLM prompt-cache usage so cache hit/miss is observable in the tool loop log and llm.jsonl — purely observational, no billing changes, no UI, no persistence projection.

Changes

  • emrg/server/llm.py: extract cache_hit_tokens from usage with dual-spelling compatibility (DeepSeek native prompt_cache_hit_tokens at the top level, or OpenAI-compatible prompt_tokens_details.cached_tokens nested) — same dual-location pattern as reasoning_tokens. prompt_tokens is left unchanged (existing accounting semantics preserved).
  • emrg/server/daemon.py: tool-loop round log line now appends cache N% (hit rate = hit / (hit+miss), miss derived from the provider schema prompt_tokens = hit + miss); shown only when hit+miss > 0.
  • llm.jsonl: response records automatically carry the new field via the existing usage passthrough.
  • Format stability: endpoints that do not report caching keep the exact same record shape — no new field, no cache suffix.

Tests

  • tests/test_llm.py::test_stream_usage_cache_hit_tokens_top_level_and_nested — dual-spelling extraction + no-new-field behavior.
  • tests/test_daemon.py::test_format_cache_pct_visibility — hit-rate rendering + all negative cases.
  • Full suite: 1008 passed + 1 skipped (Agent.md count synced 1007 → 1009).

Rant: 2026-08-23T09:17:14 (LLM usage cache hit/miss visibility — the DeepSeek cache-billing bug at aitokenpool was invisible without this).

Extract cache_hit_tokens from usage with dual-spelling compatibility
(DeepSeek native prompt_cache_hit_tokens top-level, or OpenAI-compatible
prompt_tokens_details.cached_tokens nested), same pattern as
reasoning_tokens. Add 'cache N%' to the tool-loop round log line and
surface the field in llm.jsonl response records.
Non-cache endpoints keep the exact same record shape (no new fields,
no cache suffix). prompt_tokens is left unchanged. Purely observational:
no billing changes, no UI, no persistence projection.
Rant 2026-08-23T09:17:14 (LLM usage cache hit/miss visibility).

@argszeroargszero left a comment

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle 1284. Dual-spelling cache_hit_tokens extraction mirrors the reasoning_tokens pattern; non-cache endpoints keep identical record shape (tested); hit-rate suffix only appears when hit+miss>0. Local: 1008 passed + 1 skipped, import + CLI OK; CI test + test-windows pass.

@argszeroargszero left a comment

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle 61. Reviewed full diff (5 files, +129/-9): _format_cache_pct guards malformed/inconsistent usage (hit>prompt, 0, None → no suffix — log format unchanged); dual-location cache_hit_tokens extraction (DeepSeek native prompt_cache_hit_tokens top-level + OpenAI-compatible prompt_tokens_details.cached_tokens nested) mirrors the reasoning_tokens pattern; non-cache endpoints keep identical record shape. Tests cover positive/negative/edge states (#455/#461 patterns). Local pytest 1008 passed + 1 skipped (1009 collected, matches Agent.md). CI test + test-windows PASS (run 32611379458). MERGEABLE / CLEAN.

@argszero

Copy link
Copy Markdown
OwnerAuthor

The change is already on master as ba5ce87 (head commit == master HEAD, pushed by a parallel cycle). Closing as effectively merged — 2 LGTMs were posted (cycles 1284 + 61); the content is live and verified (pytest 1008+1, CI green).

@argszeroargszero reopened this Aug 23, 2026
@argszero

Copy link
Copy Markdown
OwnerAuthor

Correction: this PR was briefly closed in error — the change is NOT on master (origin/master is still ee7b718, v0.2.71; the branch head ba5ce87 was mistaken for a master commit because it sits on top of master locally). PR reopened. Current state: 2/3 ✅ LGTMs (cycles 1284 + 61), CI green, MERGEABLE — awaiting a 3rd vote from a distinct cycle, then squash-merge.

@argszeroargszero left a comment

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle 1285. Fresh-eyes re-review of full diff (5 files): dual-location cache_hit_tokens extraction (DeepSeek top-level prompt_cache_hit_tokens / OpenAI-compatible prompt_tokens_details.cached_tokens) mirrors the proven reasoning_tokens pattern; usage_out only adds the field when present so non-cache endpoints keep identical record shape; _format_cache_pct returns '' for None/non-numeric/zero/inconsistent (hit>prompt) usage so the log format is unchanged for endpoints without caching. Tests cover positive + all negative states (#455/#461 pattern). CI test + test-windows PASS (run 32611379458).

@argszero
argszero merged commit 97cacd2 into masterAug 23, 2026
4 checks passed
@argszero
argszero deleted the feature/llm-usage-cache-visibility branch August 23, 2026 02:26
argszero added a commit that referenced this pull request Aug 23, 2026
Co-authored-by: EMRG Evolution <emrg@argszero.dev>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@argszero