feat(#822): warn when a run's prompts sit under the model's cache minimum - #983
Open
0frankie wants to merge 1 commit into
Open
feat(#822): warn when a run's prompts sit under the model's cache minimum#9830frankie wants to merge 1 commit into
0frankie wants to merge 1 commit into
Conversation
…imum Anthropic's minimum cacheable prefix is model-dependent (Haiku 4.5: 4096 tokens, Sonnet/Opus 4.x: 1024, Opus 5: 512), and a prompt below it silently never caches -- cache_creation_input_tokens stays 0 with no error, which is how two runs and 981k input tokens read as 'caching is dead' when the prompts were just under Haiku's bar (#822). The ledger now says so: CACHE_MIN_PROMPT_TOKENS pins the per-model minimums, UsageLedger.cache_warning() names each model whose median real prompt (failed calls and zero-token mock rows excluded) sits under its minimum with zero cache traffic, and summary() carries the string as cache_warning -- so it reaches the RunLog footer, GET /usage, and the run monitor through the existing seams. Models without a known minimum (OpenAI, mock, unlisted) are never warned about, and any cache traffic on a model silences its warning. live.ts's UsageSummary names the new field -- the emitter-driven mirror test (test_live_ts_usage_summary_mirrors_the_usage_route) requires the union of wire fields, so the type rides this PR; carry it onto prod with the next web sync, like #979/#980. The issue's second question (pad the Haiku prefix past 4096 to buy eligibility) stays open -- that's a measurement, not a warning. TDD: 5 new tests in tests/test_usage.py, each watched failing (ImportError/AttributeError) before implementation. Suites: engine 1502 passed / 2 skipped, backend 971 passed (godot scene smoke deselected -- times out in any fresh worktree, pre-existing on base). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Decision 1 of #822: make a silently-uncacheable run announce itself instead of being distinguishable only by post-hoc token arithmetic.
CACHE_MIN_PROMPT_TOKENSpins Anthropic's per-model minimum cacheable prefix (Haiku 4.5: 4096, Sonnet 5/4.x + Opus 4.x: 1024, Opus 5: 512), withcache_min_prompt_tokens()returningNonefor models without a known minimum (OpenAI, mock, unlisted) — those are never warned about.UsageLedger.cache_warning()names each model whose median real prompt sits under its minimum with zero cache traffic on the run. Failed calls (Live LLM: mid-run API failures are invisible — decide errors swallowed, failed calls never reach the ledger #745) and zero-token mock pacing rows are not prompts and don't drag the median; any cache traffic on a model silences its warning (it cleared the bar at least sometimes).summary()carries it ascache_warning(string or null) — one seam that reaches the RunLog footer,GET /usage, and the run monitor with no new plumbing, same asfailed_calls(Live LLM: mid-run API failures are invisible — decide errors swallowed, failed calls never reach the ledger #745).Sample:
prompt caching never engages for claude-haiku-4-5: median prompt 1500 tokens is under its 4096-token cache minimumWeb mirror rides along (carry onto prod)
test_live_ts_usage_summary_mirrors_the_usage_routedrives the real route and requireslive.ts'sUsageSummaryto name every wire field, so the type gainscache_warning?: string | nullin this PR — the same documented exception as #979/#980. Flag for the next main→prod web sync.Out of scope
The issue's decision 2 — padding the stable Haiku prefix past 4096 to buy cache eligibility — is a measurement experiment, deliberately not bundled with the warning.
Verification
tests/test_usage.py, each watched failing (ImportError/AttributeError) before the implementation existed.live.tsrequirement).black --checkclean.test_godot_scenes_rendertimes out in any fresh worktree (no imported.godotcache, [release] Retire restricted art before the public repo publish #876 assets absent) — pre-existing on the base commit, unrelated.Closes #822 (decision 1; decision 2 tracked in the issue).
🤖 Generated with Claude Code