Skip to content

Update LiteLLM model configurations and Ollama pull script to use latest model versions #312

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bubacoder merged 1 commit into from
Apr 12, 2026
Merged

Update LiteLLM model configurations and Ollama pull script to use latest model versions #312

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 39 additions & 9 deletions .claude/commands/update-litellm-models.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,18 @@
description: Update LiteLLM config/config.yaml to use the latest model versions from each provider
---

Update the LiteLLM model configuration to use the latest available model IDs, removing outdated versions.
Update the LiteLLM model configuration to use the latest available model IDs, removing outdated versions. Also update the Ollama pull script and sync LiteLLM's local Ollama entries.

## Config file location
## Config file locations

Read the current config via:
Read the current LiteLLM config via:
```bash
docker exec litellm cat /app/config.yaml
```

The source file is at `docker/ai/litellm/config/config.yaml`.
Source files:
- `docker/ai/litellm/config/config.yaml` — LiteLLM model list
- `scripts/get-offline-data-ollama.sh` — Ollama model pull script

## Step 1: Look up latest models per provider (run searches in parallel)

Expand All @@ -23,38 +25,66 @@ Search the web for the current model IDs for each provider present in the config
- **OpenRouter (Meta Llama)**: Search "Meta Llama latest models openrouter model IDs"
- **OpenRouter (DeepSeek)**: Search "DeepSeek latest models openrouter model IDs"
- **OpenRouter (xAI Grok)**: Search "xAI Grok latest models openrouter model IDs"
- **Ollama local (≤3B)**: Search "best 3B LLM model Ollama <year> benchmark" → check https://ollama.com/library
- **Ollama local (≤8B)**: Search "best 8B LLM model Ollama <year> benchmark" → check https://ollama.com/library

For each provider, identify:
1. The **latest stable** model ID (not preview/experimental unless that's the only option)
2. Whether the currently configured model has been **superseded** by a newer release
3. The **exact API model ID string** to use in LiteLLM params

For Ollama local models, pick the **best-performing model** in each size class based on current benchmarks (HumanEval, MMLU, etc.), not just the newest release date.

## Step 2: Determine what to change

### LiteLLM cloud models

For each model in the config, decide:
- **Update**: A newer stable version exists → update the `model_name` key and `litellm_params.model` value
- **Keep**: Still the latest stable model → no change needed
- **Remove**: Superseded by a newer model already listed in the same config (avoid duplicates)

Preserve without changes:
- Local Ollama models (`ollama_chat/...`) — these are managed separately
- The overall YAML structure, comments, and provider groupings
- `api_key` and `api_base` references

## Step 3: Update the file
### Ollama local models (`scripts/get-offline-data-ollama.sh`)

The script must always include exactly:
- **One model ≤3B** — best benchmark performer in this size class (e.g. `phi4-mini`)
- **One model ≤8B** — best benchmark performer in this size class (e.g. `qwen3:8b`)
- **Embedding models** — keep as-is unless a clearly better alternative exists
Comment on lines +53 to +56

@coderabbitai coderabbitai Bot Apr 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Fix the size-class contradiction in the update rule.

The rule says “≤3B” but the included example (phi4-mini) is 3.8B. This is internally inconsistent and will cause drift between docs and implementation.

Suggested wording fix 
-- **One model ≤3B** — best benchmark performer in this size class (e.g. `phi4-mini`)
+- **One model <4B** — best benchmark performer in this size class (e.g. `phi4-mini`)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
The script must always include exactly:
- **One model ≤3B** — best benchmark performer in this size class (e.g. `phi4-mini`)
- **One model ≤8B** — best benchmark performer in this size class (e.g. `qwen3:8b`)
- **Embedding models** — keep as-is unless a clearly better alternative exists
The script must always include exactly:
- **One model <4B** — best benchmark performer in this size class (e.g. `phi4-mini`)
- **One model ≤8B** — best benchmark performer in this size class (e.g. `qwen3:8b`)
- **Embedding models** — keep as-is unless a clearly better alternative exists
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.claude/commands/update-litellm-models.md around lines 53 - 56, The rule
"One model ≤3B" conflicts with the example "phi4-mini" (which is 3.8B); update
the wording so the size-class and the example match: either change the size
bound to "≤4B" (and keep phi4-mini) or replace the example with an actual ≤3B
model; adjust the line that currently reads "One model ≤3B — best benchmark
performer in this size class (e.g. `phi4-mini`)" accordingly and ensure other
examples like `qwen3:8b` remain consistent.


Update both the pull tags and the inline comments with model size.

### LiteLLM Ollama entries sync

After updating `get-offline-data-ollama.sh`, update the `ollama-local-*` entries in `config.yaml` to match:
- `ollama_chat/<model>` must reflect the exact tag used in the pull script
- Update the comment line above each entry (e.g. `# Local model - Phi 4 Mini (3.8B)`)
- Do **not** change `api_base` or `api_key` references

Write the updated config using:
## Step 3: Update the files

Write the updated LiteLLM config using:
```bash
tee "$(git rev-parse --show-toplevel)/docker/ai/litellm/config/config.yaml" > /dev/null << 'EOF'
<updated content>
EOF
```

Also update `router_settings.fallbacks` to reflect any renamed models.
Write the updated Ollama script using:
```bash
tee "$(git rev-parse --show-toplevel)/scripts/get-offline-data-ollama.sh" > /dev/null << 'EOF'
<updated content>
EOF
```

Also update `router_settings.fallbacks` in `config.yaml` to reflect any renamed models.

## Step 4: Verify

Confirm the file was written correctly:
Confirm the LiteLLM config was written correctly:
```bash
docker exec litellm cat /app/config.yaml
```
Comment on lines 85 to 90

@coderabbitai coderabbitai Bot Apr 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Add required lint checks to the verification step.

Verification should include ShellCheck for the updated script and YAML lint/validation for config changes, not only container readback.

Suggested verification additions 
 ## Step 4: Verify

 Confirm the LiteLLM config was written correctly:
 ```bash
 docker exec litellm cat /app/config.yaml

+Run required lint checks:
+bash +shellcheck "$(git rev-parse --show-toplevel)/scripts/get-offline-data-ollama.sh" +yamllint "$(git rev-parse --show-toplevel)/docker/ai/litellm/config/config.yaml" +

</details>

As per coding guidelines `**/*.sh`: "Shell scripts must pass ShellCheck validation as part of pre-commit checks" and `**/*.{yaml,yml}`: "YAML files must pass linting and validation as part of pre-commit checks".

<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

```suggestion
## Step 4: Verify

Confirm the LiteLLM config was written correctly:
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.claude/commands/update-litellm-models.md around lines 85 - 90, Update the
"Step 4: Verify" section to add lint checks in addition to the container
readback: run ShellCheck against the updated script
scripts/get-offline-data-ollama.sh and run YAML lint/validation against
docker/ai/litellm/config/config.yaml (or the config file referenced in the doc)
so verification includes shellcheck and yamllint as required by the repo
pre-commit rules; mention both commands in the verification step and ensure
paths point to the repository root as in the suggested snippet.

Expand Down
28 changes: 17 additions & 11 deletions docker/ai/litellm/config/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,14 +41,14 @@ model_list:
# Create API key: https://platform.openai.com/api-keys
# Buy credits: https://platform.openai.com/settings/organization/billing/overview

- model_name: gpt-5
- model_name: gpt-5.4
litellm_params:
model: openai/gpt-5
model: openai/gpt-5.4
api_key: "os.environ/OPENAI_API_KEY"

- model_name: gpt-5-mini
- model_name: gpt-5.4-mini
litellm_params:
model: openai/gpt-5-mini
model: openai/gpt-5.4-mini
api_key: "os.environ/OPENAI_API_KEY"

# --- Via Google
Expand Down Expand Up @@ -109,20 +109,26 @@ model_list:
model: openrouter/deepseek/deepseek-r1
api_key: "os.environ/OPENROUTER_API_KEY"

# https://openrouter.ai/x-ai/grok-code-fast-1
- model_name: grok-code-fast-1
# https://openrouter.ai/x-ai/grok-4.1-fast
- model_name: grok-4.1-fast
litellm_params:
model: openrouter/x-ai/grok-code-fast-1
model: openrouter/x-ai/grok-4.1-fast
api_key: "os.environ/OPENROUTER_API_KEY"

# https://openrouter.ai/z-ai/glm-5.1
- model_name: glm-5.1
litellm_params:
model: openrouter/z-ai/glm-5.1
api_key: "os.environ/OPENROUTER_API_KEY"

# --- Local models (Ollama)
# LiteLLM provider: https://docs.litellm.ai/docs/providers/ollama#using-ollama-apichat
# Models: https://ollama.com/search

# Local model - Phi 3.5
# Local model - Phi 4 Mini (3.8B)
- model_name: ollama-local-phi
litellm_params:
model: ollama_chat/phi3.5:3.8b
model: ollama_chat/phi4-mini
api_base: "os.environ/LOCAL_OLLAMA_API_BASE"
api_key: "none"

Expand All @@ -135,9 +141,9 @@ model_list:

router_settings:
fallbacks:
- gpt-5:
- gpt-5.4:
- gpt-4.1
- gpt-5-mini:
- gpt-5.4-mini:
- gpt-4.1-mini
- ollama-mac-mistral:
- ollama-local-phi
4 changes: 2 additions & 2 deletions scripts/get-offline-data-ollama.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ ollama_pull() {
}

# Large Language Models - https://ollama.com/search
ollama_pull llama3.2:latest
ollama_pull qwen2.5-coder:7b
ollama_pull phi4-mini # 3.8B - best-in-class sub-4B; 128K ctx
ollama_pull qwen3:8b # 8B - best-in-class 7-8B; hybrid thinking, top HumanEval
Comment on lines +23 to +24

@coderabbitai coderabbitai Bot Apr 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Sync the new 8B pull target with LiteLLM model routing.

qwen3:8b is pulled here, but LiteLLM config still routes the 8B slot to ollama_chat/mistral:7b-instruct (model_name: ollama-mac-mistral). That leaves the new pull target unused and the configured 8B route stale.

Suggested sync update (config side) 
-  # Local model - Mistral 7b
-  - model_name: ollama-mac-mistral
+  # Local model - Qwen3 (8B)
+  - model_name: ollama-local-qwen
     litellm_params:
-      model: ollama_chat/mistral:7b-instruct
+      model: ollama_chat/qwen3:8b
       api_base: "os.environ/REMOTE_OLLAMA_API_BASE"
       api_key: "none"

 router_settings:
   fallbacks:
-    - ollama-mac-mistral:
+    - ollama-local-qwen:
         - ollama-local-phi
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@scripts/get-offline-data-ollama.sh` around lines 23 - 24, The 8B model pulled
in scripts/get-offline-data-ollama.sh (qwen3:8b) is not used because the LiteLLM
routing still maps the 8B slot to ollama_chat/mistral:7b-instruct (model_name:
ollama-mac-mistral); update the LiteLLM config to route the 8B slot to qwen3:8b
(or rename the pull to match the configured model) by changing the 8B model
mapping that references model_name: ollama-mac-mistral to point to qwen3:8b (or
adjust the pull target to match the existing model_name) so the pulled model and
the routing are consistent.


# Embedding Models - https://ollama.com/search?c=embedding
ollama_pull nomic-embed-text
Loading