Uh oh!
There was an error while loading. Please reload this page.
Add provider smokes and model preference - #379
Conversation
bussyjd
commented
Apr 27, 2026
This still makes sense — does not duplicate or conflict with #386 / #380 / #387. Two small adjustments needed before it can land alongside them; I've prepared a rebased branch with both changes ready: What changed in the rebase:
Validation against the rebase tip: Recommended ship order: #387 → #380 → #386 → swap this PR's branch for the rebased one (or merge it as-is and close this). The rebased branch will continue to apply cleanly even if the order changes; Note on |
bussyjd
commented
Apr 27, 2026
Self-update for the maintainer: I pushed a34442e on this branch — renames The - return syncOpenClawModels(cfg, u)+ return syncAgentModels(cfg, u)Both call sites (the |
The original PR moved the chosen model to the front of the LiteLLM model_list and assumed downstream resolution would honor that order. But on main, `rankModels` delegates to `model.Rank` for capability-aware ranking — cloud > local, larger local > smaller local — which silently overrode the user's pick on every stack restart, defaults refresh, or auto-configure pass that re-read the model list. This commit makes the preference load-bearing: - `internal/model/preference.go` — persist the user's pick in `$OBOL_CONFIG_DIR/preferred-model`. Survives ConfigMap rewrites, unlike model_list[0] which any auto-config path can clobber. - `internal/model/rank.go` — new `RankWithPreference(models, pref)` that returns the preferred model as primary when present, falling through to plain `Rank` when the preference is empty or stale. Preserves provider-prefix tolerance so `claude-sonnet-4-6` matches `anthropic/claude-sonnet-4-6`. - `internal/openclaw/openclaw.go` and `internal/hermes/hermes.go` — `rankModels` now takes cfg and consults the preference file; the ConfigMap reorder in `PreferModel` is kept purely as a UX hint so `obol model list` shows the preferred entry on top. - `PreferModel` writes the preference file FIRST and patches the ConfigMap second — the preference sticks even when the cluster is down. Tests: - `TestRankWithPreference_*` — five cases covering no-preference, explicit preference winning over capability, provider-prefix tolerance, stale preference falling back, empty input. - `TestPreferenceRoundTrip`, `TestPreferenceClear`, `TestPreferenceSurvivesArbitraryFiles` — persistence contract. - `TestRankModelsCapabilityWinsWithoutPreference`, `TestRankModelsHonorsExplicitPreference`, `TestRankModelsStalePreferenceFallsBackToCapability` — replace the old `TestRankModelsHonorsConfiguredOrder`, which asserted the bug itself (that rankModels returned models[0] regardless of capability). The defaults-refresh path (autoConfigureLLM) is unaffected — it patches the LiteLLM ConfigMap but never touches the preference file, so the preference survives every restart and refresh. Resolves the rank-overrides-prefer concern raised in #379 review.
a34442e to
6156195Comparebussyjd
commented
Apr 29, 2026
Rebased onto current main + addressed the substantive design issue Oisin flagged. What changedRebase: dropped The substantive fix (new commit on top: |
OisinKyne
commented
Apr 29, 2026
closing for now in favour of doing model control on the calling agent side rather than the litellm side |
Summary
obol model prefer <model-name>to move a configured LiteLLM model to the front of the preference order and sync stack-managed agent model config.Tests
bash -n flows/flow-12-agent-provider-smokes.sh flows/release-smoke.shgo test ./cmd/obol ./internal/model ./internal/openclawgo test ./... -run ^'$' -count=0Closes#378