emrg: usage-anchored auto-compact + stable system prefix for prompt cache (rants 2026-08-23T13:28:50, 13:54:14) - #946
Conversation
…s 2026-08-23T13:28:50 + 13:54:14)
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle 1/3: usage-anchored auto-compact projection (anchor from real prompt_tokens + estimate delta, stale-anchor fallback) + current_time moved out of system prefix into trailing context user message (context_refresh_interval_ms gating). Verified locally: 1018 passed + 1 skipped, import + CLI OK; CI test + test-windows green (run 32621932870). Mergeable.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle 2/3: head e69494f unchanged since cycle-1 review; CI test + test-windows still green (run 32621932870); PR mergeable. Usage-anchored auto-compact projection + stable system prefix (current_time → trailing context message, context_refresh_interval_ms gating) verified locally: 1018 passed + 1 skipped, import + CLI OK.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle 3/3: head e69494f unchanged since cycles 1-2; CI test + test-windows green (run 32621932870); PR mergeable. Local baseline 1018 passed + 1 skipped, import + CLI OK (verified in cycle 1). 3 consecutive LGTMs from different cycles reached — merging.
Uh oh!
There was an error while loading. Please reload this page.
…mpact + stable system prefix) (#947) Co-authored-by: EMRG Evolution <emrg@argszero.dev>
…switch (#1003) Dev.to reader feedback (comment 3dh3g on post 3, tracked as issue #1000): the usage anchor is keyed by session_id only, so a mid-session model switch projected the OLD provider's real prompt_tokens base + the NEW estimate delta — a mixed base that can silently miss the auto-compact gate across providers (the #946 failure mode). Fix: _handle_set_model invalidates all usage anchors (and any pending anchor-drift window) when the API model actually changes, and marks sessions so the fail-LOUD missing-anchor warning treats the switch round as a legitimate re-anchor round (next response re-anchors from the new provider's real prompt_tokens). +4 tests; Agent.md Python count 1102 -> 1106. Co-authored-by: EMRG Evolution <emrg@argszero.dev>
Two token/cost fixes inspired by the host's observations (learning from deepseek-harness):
1. Usage-anchored auto-compact (rant 2026-08-23T13:28:50)
_estimate_tokens), which systematically underprices CJK/JSON (observed 148K est vs 222K real → threshold missed, 130-220K token requests kept going uncompacted).(last_real_prompt_tokens, last_local_estimate)refreshed after every LLM round fromfinal_usage.prompt_tokens. The trigger now usesprojected = anchor_tokens + (current_estimate − anchor_estimate)— the estimator error only affects the delta, never accumulates. Stale anchors (post-compact shrink) fall back to the plain estimate; the anchor is dropped after compaction.2. Stable system prompt prefix for prompt caching (rant 2026-08-23T13:54:14)
current_timewas rendered into the system prefix and changed every request → the whole prefix (up to ~175K tokens) missed cache on every request's round 1 (cache 0% observed).current_timeremoved fromsystem.j2/prompt builder; dynamic context is now injected as a trailing user message[context] Current time: ...before the user prompt, gated by a newcontext_refresh_interval_msconfig (default 0 = always fresh; >0 freezes the snapshot for cache-friendly byte-stable prefixes).Tests: +6 (anchor projection trigger/fallback, context injection format/refresh/fresh); updated env tests (time no longer in prefix) + ws_e2e context_messages 5→6. Verified: 1018 passed + 1 skipped, import + CLI OK.