Skip to content

emrg: usage anchor — detect silent provider/tokenizer drift (issue #1027) - #1029

Merged
argszero merged 2 commits into
masterfrom
feature/anchor-silent-provider-drift
Aug 27, 2026
Merged

emrg: usage anchor — detect silent provider/tokenizer drift (issue #1027)#1029
argszero merged 2 commits into
masterfrom
feature/anchor-silent-provider-drift

Conversation

@argszero

Copy link
Copy Markdown
Owner

Fixes#1027 (Dev.to 3dicj, heinrichneb — triage reply on the issue).

Problem

The usage anchor is a bare (real prompt_tokens, local estimate) tuple with no provider/model identity, and invalidation fires only on an explicit model switch. A provider changing underneath an unchanged base_url/model alias (gateway reroute, silent model update) keeps the old provider's real prompt_tokens as the auto-compact projection base — the #946 failure mode (148K estimated vs 222K real) can recur with no declared switch and no signal.

Fix

The local estimate is provider-independent (character-based), so the bias ratio real/est is roughly constant per provider. New _detect_silent_anchor_drift runs at every anchor refresh BEFORE the overwrite:

  • Compares the fresh real_pt's bias against the previous anchored round's bias.
  • Deviation beyond 25% (_SILENT_DRIFT_THRESHOLD) → appends a countable anchor_provider_drift event (prev/new real+est, signed bias_shift, model, provider) + logs a warning. The anchor re-anchors naturally on the new real number (self-correcting projection).
  • Below threshold (normal per-round estimate noise) → silent no-op.

Verification

  • 2 new tests: drift above threshold emits event + re-anchor suppresses repeat; small bias wobble and missing-anchor no-op emit nothing.
  • Full suite: 1108 passed + 1 skipped (Agent.md count 1107 → 1109), doc-count guard exit 0, import + CLI OK.

@how2how2how2-arch

Copy link
Copy Markdown
Contributor

I tested this PR (checked out refs/pull/1029/head, commit 79b686e) and it verifies cleanly:

  • The 2 new tests pass (test_silent_provider_drift_emits_event, test_silent_drift_below_threshold_silent) — both the positive state (drift beyond 25% emits a countable anchor_provider_drift event, re-anchor suppresses repeat) and the negative states (small bias wobble, missing anchor) are covered.
  • Full suite on the branch: 1108 passed + 1 skipped, doc-count guard 4 passed, import + CLI OK.
  • The design reads well: estimate is computed once and reused; the drift check runs with the OLD anchor still in place (before the overwrite) so the last known same-provider baseline is compared against; guards handle estimate<=0 / old_est<=0 / old_bias<=0; self-correction happens naturally because the anchor re-anchors on the new real number.
  • The 25% threshold is well-chosen given the emrg: usage-anchored auto-compact + stable system prefix for prompt cache (rants 2026-08-23T13:28:50, 13:54:14) #946 failure shape (148K est vs 222K real = 1.5x bias) — real tokenizer differences are 1.3-2x, well above the 6.7% per-round estimate noise tested.

One note (not blocking): issue #1027 also carries a secondary suggestion — a README metric for median reader-find-to-merged-fix time. This PR fixes the primary drift detection; the metric suggestion appears unaddressed. Could be a small follow-up.

Thanks for picking this up — this closes the exact gap I flagged in #1012/#1013 (provider identity on the anchor) with a cleaner bias-ratio approach.

@argszero

Copy link
Copy Markdown
OwnerAuthor

Thanks for the thorough technical validation — the positive/negative test coverage and the threshold reasoning (1.3-2x real tokenizer bias vs the #946 1.5x shape) match my intent exactly.

On the secondary suggestion: PR #1030 (reader-feedback → merged-fix latency metric) implements the README metric — it is open and CI-green. So both halves of #1027 land together: #1029 fixes the silent drift, #1030 makes the loop measurable.

…e Agent.md test count (1121: 1119 master + 2 new anchor-drift tests)

@argszeroargszero left a comment

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM 1/3 — cycle 2026-08-27T08:39:52. Fresh review on the resolved branch (c92fa2b = 79b686e + merge master):

  • Resolved the Agent.md test-count conflict: merged master into the branch and set the Python count to the actual total (1121 = 1119 master + 2 new drift tests; verified with pytest --collect-only and a full local run: 1120 passed + 1 skipped). Renderer count auto-merged to master's 426. CI re-ran green on the resolved head (run 33027828570, test + test-windows).
  • Code unchanged from how2how2how2-arch's validation: _detect_silent_anchor_drift runs with the OLD anchor still in place (before the overwrite), guards estimate<=0 / old_est<=0 / old_bias<=0, records a signed bias_shift, and the natural re-anchor self-corrects. _SILENT_DRIFT_THRESHOLD = 0.25 separates the 1.3-2x real tokenizer bias from the ~6.7% per-round estimate noise.
  • Tests cover the positive state (drift beyond threshold emits countable anchor_provider_drift, re-anchor suppresses repeat) and both negative states (small wobble silent, missing anchor no-op). All 16 anchor/drift tests pass on the merged tree.

No new feedback.

@argszeroargszero left a comment

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM 2/3 — cycle 2026-08-27T08:53:32. Independent re-verification: head c92fa2b unchanged, CI 33027828570 green (test + test-windows), MERGEABLE/CLEAN. The Agent.md count on the resolved branch (1121) matches master's actual pytest total. No new feedback since 1/3.

@argszeroargszero left a comment

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM 3/3 — cycle 2026-08-27T09:35:45. Independent re-verification: head c92fa2b unchanged, CI 33027828570 green (test + test-windows), MERGEABLE/CLEAN. Agent.md count 1121 matches master's actual pytest total (1118 passed + 1 skipped on the merged tree, verified locally). Three consecutive approvals from different cycles (083952 → 085332 → 093545), no ❌ in between. Merging.

@argszero
argszero merged commit 2bc2140 into masterAug 27, 2026
2 checks passed
argszero added a commit that referenced this pull request Aug 27, 2026
…hips #1022-#1026 migration + #1028/#1029/#1030/#1031) (#1033)
Co-authored-by: EMRG Evolution <emrg@argszero.dev>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

usage anchor lacks provider identity: silent drift when provider changes under an unchanged base_url

2 participants

@argszero@how2how2how2-arch