Update LiteLLM model configurations and Ollama pull script to use latest model versions - #312
Merged
Conversation
…est model versions
Contributor
WalkthroughDocumentation and configuration updates to LiteLLM model management: expanded update procedures to include Ollama script synchronization, replaced model identifiers for OpenAI/OpenRouter models, added GLM-5.1 entry, replaced local Ollama models in configuration and pull script, and updated fallback references. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.claude/commands/update-litellm-models.md:
- Around line 53-56: The rule "One model ≤3B" conflicts with the example
"phi4-mini" (which is 3.8B); update the wording so the size-class and the
example match: either change the size bound to "≤4B" (and keep phi4-mini) or
replace the example with an actual ≤3B model; adjust the line that currently
reads "One model ≤3B — best benchmark performer in this size class (e.g.
`phi4-mini`)" accordingly and ensure other examples like `qwen3:8b` remain
consistent.
- Around line 85-90: Update the "Step 4: Verify" section to add lint checks in
addition to the container readback: run ShellCheck against the updated script
scripts/get-offline-data-ollama.sh and run YAML lint/validation against
docker/ai/litellm/config/config.yaml (or the config file referenced in the doc)
so verification includes shellcheck and yamllint as required by the repo
pre-commit rules; mention both commands in the verification step and ensure
paths point to the repository root as in the suggested snippet.
In `@scripts/get-offline-data-ollama.sh`:
- Around line 23-24: The 8B model pulled in scripts/get-offline-data-ollama.sh
(qwen3:8b) is not used because the LiteLLM routing still maps the 8B slot to
ollama_chat/mistral:7b-instruct (model_name: ollama-mac-mistral); update the
LiteLLM config to route the 8B slot to qwen3:8b (or rename the pull to match the
configured model) by changing the 8B model mapping that references model_name:
ollama-mac-mistral to point to qwen3:8b (or adjust the pull target to match the
existing model_name) so the pulled model and the routing are consistent.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: ecba4e4e-1301-475c-9dc9-1d5e9b211750
📒 Files selected for processing (3)
.claude/commands/update-litellm-models.mddocker/ai/litellm/config/config.yamlscripts/get-offline-data-ollama.sh
Comment on lines
+53
to
+56
| The script must always include exactly: | ||
| - **One model ≤3B** — best benchmark performer in this size class (e.g. `phi4-mini`) | ||
| - **One model ≤8B** — best benchmark performer in this size class (e.g. `qwen3:8b`) | ||
| - **Embedding models** — keep as-is unless a clearly better alternative exists |
Contributor
There was a problem hiding this comment.
Fix the size-class contradiction in the update rule.
The rule says “≤3B” but the included example (phi4-mini) is 3.8B. This is internally inconsistent and will cause drift between docs and implementation.
Suggested wording fix
-- **One model ≤3B** — best benchmark performer in this size class (e.g. `phi4-mini`)
+- **One model <4B** — best benchmark performer in this size class (e.g. `phi4-mini`)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| The script must always include exactly: | |
| - **One model ≤3B** — best benchmark performer in this size class (e.g. `phi4-mini`) | |
| - **One model ≤8B** — best benchmark performer in this size class (e.g. `qwen3:8b`) | |
| - **Embedding models** — keep as-is unless a clearly better alternative exists | |
| The script must always include exactly: | |
| - **One model <4B** — best benchmark performer in this size class (e.g. `phi4-mini`) | |
| - **One model ≤8B** — best benchmark performer in this size class (e.g. `qwen3:8b`) | |
| - **Embedding models** — keep as-is unless a clearly better alternative exists |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.claude/commands/update-litellm-models.md around lines 53 - 56, The rule
"One model ≤3B" conflicts with the example "phi4-mini" (which is 3.8B); update
the wording so the size-class and the example match: either change the size
bound to "≤4B" (and keep phi4-mini) or replace the example with an actual ≤3B
model; adjust the line that currently reads "One model ≤3B — best benchmark
performer in this size class (e.g. `phi4-mini`)" accordingly and ensure other
examples like `qwen3:8b` remain consistent.
Comment on lines
85
to
90
| ## Step 4: Verify | ||
|
|
||
| Confirm the file was written correctly: | ||
| Confirm the LiteLLM config was written correctly: | ||
| ```bash | ||
| docker exec litellm cat /app/config.yaml | ||
| ``` |
Contributor
There was a problem hiding this comment.
Add required lint checks to the verification step.
Verification should include ShellCheck for the updated script and YAML lint/validation for config changes, not only container readback.
Suggested verification additions
## Step 4: Verify
Confirm the LiteLLM config was written correctly:
```bash
docker exec litellm cat /app/config.yaml+Run required lint checks:
+bash +shellcheck "$(git rev-parse --show-toplevel)/scripts/get-offline-data-ollama.sh" +yamllint "$(git rev-parse --show-toplevel)/docker/ai/litellm/config/config.yaml" +
</details>
As per coding guidelines `**/*.sh`: "Shell scripts must pass ShellCheck validation as part of pre-commit checks" and `**/*.{yaml,yml}`: "YAML files must pass linting and validation as part of pre-commit checks".
<!-- suggestion_start -->
<details>
<summary>📝 Committable suggestion</summary>
> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
```suggestion
## Step 4: Verify
Confirm the LiteLLM config was written correctly:
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.claude/commands/update-litellm-models.md around lines 85 - 90, Update the
"Step 4: Verify" section to add lint checks in addition to the container
readback: run ShellCheck against the updated script
scripts/get-offline-data-ollama.sh and run YAML lint/validation against
docker/ai/litellm/config/config.yaml (or the config file referenced in the doc)
so verification includes shellcheck and yamllint as required by the repo
pre-commit rules; mention both commands in the verification step and ensure
paths point to the repository root as in the suggested snippet.
Comment on lines
+23
to
+24
| ollama_pull phi4-mini # 3.8B - best-in-class sub-4B; 128K ctx | ||
| ollama_pull qwen3:8b # 8B - best-in-class 7-8B; hybrid thinking, top HumanEval |
Contributor
There was a problem hiding this comment.
Sync the new 8B pull target with LiteLLM model routing.
qwen3:8b is pulled here, but LiteLLM config still routes the 8B slot to ollama_chat/mistral:7b-instruct (model_name: ollama-mac-mistral). That leaves the new pull target unused and the configured 8B route stale.
Suggested sync update (config side)
- # Local model - Mistral 7b
- - model_name: ollama-mac-mistral
+ # Local model - Qwen3 (8B)
+ - model_name: ollama-local-qwen
litellm_params:
- model: ollama_chat/mistral:7b-instruct
+ model: ollama_chat/qwen3:8b
api_base: "os.environ/REMOTE_OLLAMA_API_BASE"
api_key: "none"
router_settings:
fallbacks:
- - ollama-mac-mistral:
+ - ollama-local-qwen:
- ollama-local-phi🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@scripts/get-offline-data-ollama.sh` around lines 23 - 24, The 8B model pulled
in scripts/get-offline-data-ollama.sh (qwen3:8b) is not used because the LiteLLM
routing still maps the 8B slot to ollama_chat/mistral:7b-instruct (model_name:
ollama-mac-mistral); update the LiteLLM config to route the 8B slot to qwen3:8b
(or rename the pull to match the configured model) by changing the 8B model
mapping that references model_name: ollama-mac-mistral to point to qwen3:8b (or
adjust the pull target to match the existing model_name) so the pulled model and
the routing are consistent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
Release Notes
New Features
Updates