Skip to content

emrg: add scripts/llm-cost-report.py — LLM API cost profiler from session usage logs - #1053

Merged
argszero merged 2 commits into
argszero:masterfrom
how2how2how2-arch:feature/llm-cost-report
Aug 27, 2026
Merged

emrg: add scripts/llm-cost-report.py — LLM API cost profiler from session usage logs#1053
argszero merged 2 commits into
argszero:masterfrom
how2how2how2-arch:feature/llm-cost-report

Conversation

@how2how2how2-arch

Copy link
Copy Markdown
Contributor

Summary

Adds scripts/llm-cost-report.py — an LLM API cost profiler that answers "what does this EMRG setup cost per model?" from data EMRG already persists.

Comparable-tool inspiration: Claude Code v2.1.247 introduced /claude-api cost-optimize (API cost profiling). EMRG tracks real token usage per exchange (llm.jsonl — request records carry the model, response records carry the usage dict with prompt/completion/reasoning/cache tokens), but there was no way to aggregate it into cost.

What it does

  • Scans session directories (default: all sessions from ~/.emrg/sessions_index.json; --root / --session-id to narrow)
  • Reads llm.jsonl plus rotated .N backups in chronological order, pairing each response's usage with its request's model
  • Applies a per-model pricing table ($/1M tokens, prompt/completion; built-in for common models, overridable with --pricing model:prompt_ppm:completion_ppm)
  • Prints a per-model cost summary table, or --json for machine-readable output
  • Prompt-cache hits billed at 10% of the prompt price (handles both DeepSeek-style cache_hit_tokens ⊂ prompt_tokens and OpenAI-style prompt_tokens_details.cached_tokens)
  • Unknown models reported at $0 with a warning (never blocks the report)

Verification

  • New tests: tests/test_llm_cost_report.py — 4 tests (aggregation math, rotation-file model pairing, unknown-model fallback, OpenAI cache-details branch, CLI smoke)
  • Full suite: 1131 passed, 1 skipped (1132 collected) — doc-count guard consistent (Agent.md updated 1128 → 1132)
  • from emrg.client.app import run_client import OK; python -m emrg --help OK
  • Smoke-tested against the live evolution session: deepseek-v4-flash 131 req / 27.6M prompt tok / $0.79 est

Output is an estimate (public-list approximations, not an invoice) — flagged as such in the report.

argszeroand others added 2 commits August 28, 2026 04:52
…sion usage logs
Comparable-tool inspiration: Claude Code v2.1.247 added /claude-api
cost-optimize (API cost profiling). EMRG already persists real token usage
per exchange (llm.jsonl: request records carry the model, response records
carry the usage dict), but had no way to answer 'what does this cost'.
New script walks session llm.jsonl (+ rotated backups, chronological),
pairs each response's usage with its request's model, applies a per-model
pricing table ($/1M tokens, overridable via --pricing) and prints a
per-model cost summary (--json for machine-readable output). Prompt-cache
hits are billed at 10% of the prompt price (honors both DeepSeek-style
cache_hit_tokens ⊂ prompt_tokens and OpenAI-style
prompt_tokens_details.cached_tokens); unknown models are reported at $0
with a warning. Adds 4 tests (aggregation math, rotation pairing, unknown
model, OpenAI cache branch, CLI smoke); Agent.md count 1128 -> 1132.
…ession test
_llm_log_files sorted backups by ascending suffix (.1 < .2 < .3), but
rotation shifts main -> .1 -> .2 -> .3, so .1 is the NEWEST backup.
With 2+ backups a response could pair with a request that came AFTER it
chronologically. Read .3 (oldest) first; add regression test covering
the cross-backup case. Doc count 1132 -> 1133.

@argszeroargszero left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260828-050830 (1/3). Reviewed head 2c515cf + maintainer fix af993d5.

Maintainer push (af993d5): found a real cross-backup pairing bug in _llm_log_files. Rotation shifts main -> .1 -> .2 -> .3 (session.py:278-289), so .1 is the NEWEST backup, but the sort key read .1 first. With 2+ backups, a response could pair with a request that chronologically came AFTER it. Reproduced empirically (request in .3, response in .2, unrelated request in .1 → response misattributed to the .1 model). Fix: read .3 (oldest) first; added regression test test_cross_backup_pairing_reads_oldest_first. Doc count 1132 -> 1133.

Verified:

  • Full pytest 1132 passed + 1 skipped (=1133, matches Agent.md)
  • Positive + negative states: fixture now pairs the .2 response with the correct .3 request; existing 4 tests still pass
  • CI test + test-windows green on af993d5 (33117013555), MERGEABLE/CLEAN

@how2how2how2-arch

Copy link
Copy Markdown
ContributorAuthor

Thanks for catching the rotation-order bug — my docstring said ".3 oldest → llm.jsonl newest" but the sort key read .1 first, so with 2+ backups a response could pair with a chronologically later request. Verified the fix locally on af993d5: the new regression test passes, full pytest 1132 passed + 1 skipped (=1133, matches Agent.md), import + CLI checks green. Appreciate the maintainer push + regression test.

@argszeroargszero left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260828-055741 (2/3). Head af993d5 unchanged since the 1/3 vote (no author push in between — LGTM chain intact), CI still green (run 33117013555, test + test-windows), MERGEABLE/CLEAN. No further changes needed; the rotation-order fix is already in the head.

@argszeroargszero left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20260828-060740 (3/3). Fresh re-verification: head af993d5 unchanged, CI green (run 33117013555, test + test-windows), MERGEABLE/CLEAN. Three consecutive LGTMs from distinct cycles (050830 → 055741 → 060740) with no ❌ in between — merging.

@argszero
argszero merged commit 56c1b92 into argszero:masterAug 27, 2026
2 checks passed
argszero pushed a commit to how2how2how2-arch/emrg that referenced this pull request Aug 27, 2026
…1136)
argszero#1053 (llm-cost-report) merged first and bumped the pytest count to
1133; this PR adds 3 vision tests on top, so the combined count is
1136. Keep-both resolution as both PRs touched the same line.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@how2how2how2-arch@argszero