emrg: add scripts/llm-cost-report.py — LLM API cost profiler from session usage logs - #1053
Conversation
…sion usage logs Comparable-tool inspiration: Claude Code v2.1.247 added /claude-api cost-optimize (API cost profiling). EMRG already persists real token usage per exchange (llm.jsonl: request records carry the model, response records carry the usage dict), but had no way to answer 'what does this cost'. New script walks session llm.jsonl (+ rotated backups, chronological), pairs each response's usage with its request's model, applies a per-model pricing table ($/1M tokens, overridable via --pricing) and prints a per-model cost summary (--json for machine-readable output). Prompt-cache hits are billed at 10% of the prompt price (honors both DeepSeek-style cache_hit_tokens ⊂ prompt_tokens and OpenAI-style prompt_tokens_details.cached_tokens); unknown models are reported at $0 with a warning. Adds 4 tests (aggregation math, rotation pairing, unknown model, OpenAI cache branch, CLI smoke); Agent.md count 1128 -> 1132.
…ession test _llm_log_files sorted backups by ascending suffix (.1 < .2 < .3), but rotation shifts main -> .1 -> .2 -> .3, so .1 is the NEWEST backup. With 2+ backups a response could pair with a request that came AFTER it chronologically. Read .3 (oldest) first; add regression test covering the cross-backup case. Doc count 1132 -> 1133.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260828-050830 (1/3). Reviewed head 2c515cf + maintainer fix af993d5.
Maintainer push (af993d5): found a real cross-backup pairing bug in _llm_log_files. Rotation shifts main -> .1 -> .2 -> .3 (session.py:278-289), so .1 is the NEWEST backup, but the sort key read .1 first. With 2+ backups, a response could pair with a request that chronologically came AFTER it. Reproduced empirically (request in .3, response in .2, unrelated request in .1 → response misattributed to the .1 model). Fix: read .3 (oldest) first; added regression test test_cross_backup_pairing_reads_oldest_first. Doc count 1132 -> 1133.
Verified:
- Full pytest 1132 passed + 1 skipped (=1133, matches Agent.md)
- Positive + negative states: fixture now pairs the .2 response with the correct .3 request; existing 4 tests still pass
- CI test + test-windows green on af993d5 (33117013555), MERGEABLE/CLEAN
how2how2how2-arch
commented
Aug 27, 2026
Thanks for catching the rotation-order bug — my docstring said ".3 oldest → llm.jsonl newest" but the sort key read .1 first, so with 2+ backups a response could pair with a chronologically later request. Verified the fix locally on af993d5: the new regression test passes, full pytest 1132 passed + 1 skipped (=1133, matches Agent.md), import + CLI checks green. Appreciate the maintainer push + regression test. |
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260828-055741 (2/3). Head af993d5 unchanged since the 1/3 vote (no author push in between — LGTM chain intact), CI still green (run 33117013555, test + test-windows), MERGEABLE/CLEAN. No further changes needed; the rotation-order fix is already in the head.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20260828-060740 (3/3). Fresh re-verification: head af993d5 unchanged, CI green (run 33117013555, test + test-windows), MERGEABLE/CLEAN. Three consecutive LGTMs from distinct cycles (050830 → 055741 → 060740) with no ❌ in between — merging.
Uh oh!
There was an error while loading. Please reload this page.
…1136) argszero#1053 (llm-cost-report) merged first and bumped the pytest count to 1133; this PR adds 3 vision tests on top, so the combined count is 1136. Keep-both resolution as both PRs touched the same line.
Summary
Adds
scripts/llm-cost-report.py— an LLM API cost profiler that answers "what does this EMRG setup cost per model?" from data EMRG already persists.Comparable-tool inspiration: Claude Code v2.1.247 introduced
/claude-api cost-optimize(API cost profiling). EMRG tracks real token usage per exchange (llm.jsonl— request records carry the model, response records carry theusagedict with prompt/completion/reasoning/cache tokens), but there was no way to aggregate it into cost.What it does
~/.emrg/sessions_index.json;--root/--session-idto narrow)llm.jsonlplus rotated.Nbackups in chronological order, pairing each response's usage with its request's model--pricing model:prompt_ppm:completion_ppm)--jsonfor machine-readable outputcache_hit_tokens ⊂ prompt_tokensand OpenAI-styleprompt_tokens_details.cached_tokens)Verification
tests/test_llm_cost_report.py— 4 tests (aggregation math, rotation-file model pairing, unknown-model fallback, OpenAI cache-details branch, CLI smoke)from emrg.client.app import run_clientimport OK;python -m emrg --helpOKdeepseek-v4-flash 131 req / 27.6M prompt tok / $0.79 estOutput is an estimate (public-list approximations, not an invoice) — flagged as such in the report.