Skip to content

emrg: daemon: add provider/model identity to usage-anchor stats (issue #1011) - #1012

Closed
how2how2how2-arch wants to merge 1 commit into
argszero:masterfrom
how2how2how2-arch:feature/usage-anchor-stats-provider-identity
Closed

emrg: daemon: add provider/model identity to usage-anchor stats (issue #1011)#1012
how2how2how2-arch wants to merge 1 commit into
argszero:masterfrom
how2how2how2-arch:feature/usage-anchor-stats-provider-identity

Conversation

@how2how2how2-arch

Copy link
Copy Markdown
Contributor

Summary

Fixes issue #1011 (from Dev.to post-3 discussion, reader heinnebich): usage-anchor stats now carry provider identity, so anchor_loss / anchor_drift events are attributable, not just countable.

Since #1003, a mid-session model switch deliberately invalidates usage anchors. When the anchor-less window later re-anchors, the drift measurement needs to know which provider went silent at loss time vs which provider re-anchored — otherwise a cross-provider loss window appears in the drift file but cannot be attributed to either provider.

Changes

  • emrg/server/daemon.py:
    • _missing_anchor_est grows from a 2-tuple (estimated, iso_ts) to a 3-tuple (estimated, iso_ts, model), storing the provider/model active at loss time.
    • anchor_loss events now include a provider field (model name at loss time).
    • anchor_drift events now include provider_at_loss (from the stored tuple) and provider_after (model active at re-anchor) — both ends of the window are attributable.
  • tests/test_daemon.py: updated the 3-tuple in the switch-invalidation test and added provider assertions to the loss/drift tests (default gpt-4o-mini).

Verification

  • Targeted usage-anchor tests: 8/8 pass
  • Full suite: 1105 passed, 1 skipped
  • Import + CLI smoke: OK

@argszero

Copy link
Copy Markdown
Owner

Superseded by PR #1013 (same feature, submitted ~2 min later — parallel instances raced on issue #1011).

I reviewed both implementations against the issue and kept #1013 as canonical:

  1. Provider identity — emrg: usage-anchor stats — attach provider/model identity to loss/drift events (#1011) #1013 derives the provider from the LLM base_url hostname via _provider_slug (api.openai.com / api.deepseek.com / localhost), so provider and model are two distinct fields. emrg: daemon: add provider/model identity to usage-anchor stats (issue #1011) #1012 labels the model name as 'provider', which cannot distinguish two providers hosting the same model (or local vs remote), and cannot attribute 'WHICH provider went silent' when the model name is shared.
  2. Both ends of the window — emrg: usage-anchor stats — attach provider/model identity to loss/drift events (#1011) #1013 emits loss_model/loss_provider (who went silent) plus model/provider (who re-anchored) on anchor_drift; emrg: daemon: add provider/model identity to usage-anchor stats (issue #1011) #1012 has provider_at_loss/provider_after but only as model names.
  3. Regression coverage — emrg: usage-anchor stats — attach provider/model identity to loss/drift events (#1011) #1013 adds test_usage_anchor_cross_provider_window_attributable (deepseek → openai switch, every event names its provider), the exact scenario from the issue. emrg: daemon: add provider/model identity to usage-anchor stats (issue #1011) #1012 extends existing assertions only.

Both implementations are sound; #1013 is the strict superset with the cross-provider test. Closing this one to avoid a merge collision on the same code.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@how2how2how2-arch@argszero