Configure gated Ollama routing and set llm-routing-ollama as free tier fallback

README.md

@@ -141,16 +141,33 @@ sequenceDiagram
  end
 
  else Route = ollama (llm-routing-ollama, auto-ollama, auto-agy-ollama chain)
- Note over Router: Proxy to LiteLLM as ollama-deepseek-v4-pro
+ Note over Router: Proxy to LiteLLM as ollama-deepseek-v4-pro / -flash
  Router->>Proxy: POST /v1/chat/completions (model=ollama-deepseek-v4-pro)
  Proxy->>Provider: Call api.ollama.com (ollama_chat provider)
  alt Ollama Succeeds
  Provider-->>Proxy: deepseek-v4-pro response
  Proxy-->>Router: Return response
- Router-->>Client: Return chat completion
+ Router-->>Client: Return response
  else Ollama fails (rate-limited / unavailable)
- Note over Proxy: Fallback: ollama-deepseek-v4-pro → agent-advanced-core
- Proxy->>Provider: Call OpenRouter (agent-advanced-core tier)
+ Provider-->>Proxy: Error / HTTP 429
+ Proxy-->>Router: Error Response
+ Note over Router: Activate 5-min router-side Ollama cooldown
+ alt Model = auto-ollama / auto-agy-ollama (triage-gated)
+ Note over Router: Catch error → Fall back to free tier
+ Router->>Proxy: POST /v1/chat/completions (model=original_target_model)
+ Proxy->>Provider: Call OpenRouter (free tier cascade)
+ Provider-->>Proxy: Response
+ Proxy-->>Router: Response
+ Router-->>Client: Return response
+ else Model = llm-routing-ollama (direct / fallback chain)
+ Note over Router: Return 429 (cooldown active)
+ Router-->>Proxy: HTTP 429 (Ollama cooled down)
+ Note over Proxy: Skip llm-routing-ollama → cascade to openrouter-auto
+ Proxy->>Provider: Call OpenRouter (openrouter-auto)
+ Provider-->>Proxy: Response
+ Proxy-->>Router: Response
+ Router-->>Client: Return response
+ end
  end
 
  else Route = LiteLLM (all other models)
@@ -180,11 +197,11 @@ The gateway supports multiple routing modes controlled by the `model` field:
 | Model | Behavior |
 |-------|----------|
 | `llm-routing-auto-free` | **Full classifier pipeline** → routes to best free tier. Recommended default. |
-| `llm-routing-auto-agy` | **Classifier + agy (gated)**: runs classifier, tries agy only if classified as advanced. |
-| `llm-routing-auto-ollama` | **Classifier + Ollama (gated)**: runs classifier, tries Ollama only if advanced. |
-| `llm-routing-auto-agy-ollama` | **Classifier → agy → ollama (gated)**: runs classifier, chains agy then Ollama only if advanced. |
+| `llm-routing-auto-agy` | **Classifier + agy (gated)**: runs classifier, tries agy only if classified as advanced/reasoning. |
+| `llm-routing-auto-ollama` | **Classifier + Ollama (gated)**: runs classifier, reasoning & advanced → `ollama-deepseek-v4-pro`, complex → `ollama-deepseek-v4-flash`, below (medium/simple) → bypasses Ollama to LiteLLM free tiers. |
+| `llm-routing-auto-agy-ollama` | **Classifier → agy → ollama (gated)**: runs classifier, chains agy then Ollama only if advanced/reasoning. |
 | `llm-routing-agy` | **Direct agy**: skips classifier, agy proxy (Gemini/Claude) → LiteLLM fallback. |
-| `llm-routing-ollama` | **Direct Ollama**: skips classifier, Ollama deepseek-v4-pro → LiteLLM fallback. |
+| `llm-routing-ollama` | **Gated Ollama**: runs classifier, reasoning & advanced → `ollama-deepseek-v4-pro`, complex & below → `ollama-deepseek-v4-flash`. |
 | `agent-simple-core` / `agent-medium-core` / `agent-complex-core` / `agent-reasoning-core` / `agent-advanced-core` | **Direct routing**: bypasses classifier, goes straight to LiteLLM with that tier name. |
 | Anything else | Returns HTTP 400 with the list of available models |
 
@@ -238,10 +255,10 @@ Exposes the entry endpoint (`http://localhost:5000/v1`) and evaluates prompt com
 |:---|---:|:---|:---|
 | `llm-routing-auto-free` | ✅ | — | LiteLLM with classified tier | 256K |
 | `llm-routing-auto-agy` | ✅ | agy (gated: reasoning → gemini-3.5-flash, advanced → gemini-3.5-flash → claude-opus-4.6) | LiteLLM with classified tier | 256K |
-| `llm-routing-auto-ollama` | ✅ | Ollama (gated: reasoning → deepseek-v4-flash, advanced → deepseek-v4-pro) | LiteLLM agent-advanced-core | 256K |
+| `llm-routing-auto-ollama` | ✅ | Ollama (gated: reasoning & advanced → ollama-deepseek-v4-pro, complex → ollama-deepseek-v4-flash, below → bypass) | LiteLLM with classified tier | 256K |
 | `llm-routing-auto-agy-ollama` | ✅ | agy → Ollama (gated: reasoning/advanced only) | LiteLLM with classified tier | 256K |
 | `llm-routing-agy` | ❌ | agy (Gemini/Claude) — unconditional | LiteLLM agent-advanced-core | 256K |
-| `llm-routing-ollama` | ❌ | Ollama deepseek-v4-pro — unconditional | LiteLLM agent-advanced-core | 256K |
+| `llm-routing-ollama` | ✅ | Ollama (gated: reasoning & advanced → ollama-deepseek-v4-pro, complex & below → ollama-deepseek-v4-flash) | LiteLLM agent-advanced-core / agent-reasoning-core | 256K |
 | `agent-advanced-core` | ❌ | — | LiteLLM openrouter-auto |
 | `agent-reasoning-core` | ❌ | — | LiteLLM fallback chain |
 | `agent-complex-core` | ❌ | — | LiteLLM fallback chain |
@@ -259,14 +276,62 @@ Orchestrates routing fallback chains, Redis caching, and telemetry callbacks:
  - `embedding_model: "local-nomic-embed"` — uses the local nomic-embed model (no API costs)
  - `collection_name: "litellm_semantic_cache"` — stores embeddings for similarity-based cache lookups
 - **Cascading Fallback Chains** (configured in `litellm_settings.fallbacks`):
- Each tier escalates through increasingly capable models. All chains end at `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`). `local-qwen-3.6` (35B) was disabled 2026-06-08 to free 23GB RAM/GTT.
- - **`agent-simple-core`**: medium-core → complex-core → reasoning-core → advanced-core → `openrouter-auto`
- - **`agent-medium-core`**: complex-core → reasoning-core → advanced-core → `openrouter-auto`
- - **`agent-complex-core`**: reasoning-core → advanced-core → `openrouter-auto`
- - **`agent-reasoning-core`**: advanced-core → `openrouter-auto`
- - **`agent-advanced-core`**: `openrouter-auto`
- - **`ollama-deepseek-v4-pro`**: advanced-core → `openrouter-auto`
- All tiers ultimately land on the local Ryzen APU MoE (`qwen-35b-q4ks` via llama-server on :8080) as the final safety net.
+ Each tier escalates through increasingly capable free models, then paid local/remote Ollama models, and finally falls back to `openrouter-auto` (LiteLLM's internal fallback to OpenRouter `/auto`). `local-qwen-3.6` (35B) was disabled 2026-06-08 to free 23GB RAM/GTT.
+
+ ```mermaid
+ graph TD
+ %% Define styles
+ classDef simple fill:#4F46E5,stroke:#312E81,stroke-width:2px,color:#fff;
+ classDef medium fill:#7C3AED,stroke:#4C1D95,stroke-width:2px,color:#fff;
+ classDef complex fill:#DB2777,stroke:#831843,stroke-width:2px,color:#fff;
+ classDef reasoning fill:#EA580C,stroke:#7C2D12,stroke-width:2px,color:#fff;
+ classDef advanced fill:#E11D48,stroke:#881337,stroke-width:2px,color:#fff;
+ classDef premium fill:#059669,stroke:#064E3B,stroke-width:2px,color:#fff;
+ classDef auto fill:#4B5563,stroke:#1F2937,stroke-width:2px,color:#fff;
+
+ subgraph Simple["agent-simple-core Fallback Tree"]
+ S[agent-simple-core]:::simple --> SM[agent-medium-core]:::medium
+ SM --> SC[agent-complex-core]:::complex
+ SC --> SR[agent-reasoning-core]:::reasoning
+ SR --> SA[agent-advanced-core]:::advanced
+ SA --> SO1[llm-routing-ollama]:::premium
+ SO1 --> SAU[openrouter-auto]:::auto
+ end
+
+ subgraph Medium["agent-medium-core Fallback Tree"]
+ M[agent-medium-core]:::medium --> MC[agent-complex-core]:::complex
+ MC --> MR[agent-reasoning-core]:::reasoning
+ MR --> MA[agent-advanced-core]:::advanced
+ MA --> MO1[llm-routing-ollama]:::premium
+ MO1 --> MAU[openrouter-auto]:::auto
+ end
+
+ subgraph Complex["agent-complex-core Fallback Tree"]
+ C[agent-complex-core]:::complex --> CR[agent-reasoning-core]:::reasoning
+ CR --> CA[agent-advanced-core]:::advanced
+ CA --> CO1[llm-routing-ollama]:::premium
+ CO1 --> CAU[openrouter-auto]:::auto
+ end
+
+ subgraph Reasoning["agent-reasoning-core Fallback Tree"]
+ R[agent-reasoning-core]:::reasoning --> RA[agent-advanced-core]:::advanced
+ RA --> RO1[llm-routing-ollama]:::premium
+ RO1 --> RAU[openrouter-auto]:::auto
+ end
+
+ subgraph Advanced["agent-advanced-core Fallback Tree"]
+ A[agent-advanced-core]:::advanced --> AO1[llm-routing-ollama]:::premium
+ AO1 --> AAU[openrouter-auto]:::auto
+ end
+ ```
+
+ - **`agent-simple-core`**: medium-core → complex-core → reasoning-core → advanced-core → `llm-routing-ollama` → `openrouter-auto`
+ - **`agent-medium-core`**: complex-core → reasoning-core → advanced-core → `llm-routing-ollama` → `openrouter-auto`
+ - **`agent-complex-core`**: reasoning-core → advanced-core → `llm-routing-ollama` → `openrouter-auto`
+ - **`agent-reasoning-core`**: advanced-core → `llm-routing-ollama` → `openrouter-auto`
+ - **`agent-advanced-core`**: `llm-routing-ollama` → `openrouter-auto`
+ - **`llm-routing-ollama`** (classifier-gated proxy): `reasoning & advanced` → `ollama-deepseek-v4-pro`, `complex & below` → `ollama-deepseek-v4-flash`. Note: Ollama cooldowns are managed by the triage router internally (5-minute window on failure); during cooldown the router returns 429 immediately so LiteLLM skips to `openrouter-auto`.
+ All tiers ultimately land on OpenRouter auto/free model pools or the local Speculative MoE when enabled.
 *Note: Premium routing is controlled by the model name, not by the tier. `llm-routing-agy` and `llm-routing-auto-agy` trigger the agy proxy (Google/Claude via Cloud Code Assist) — but auto models only trigger agy if the classifier returns `agent-advanced-core`. `llm-routing-ollama` and `llm-routing-auto-ollama` route through Ollama.com (deepseek-v4-pro via LiteLLM's ollama_chat provider) — same gating for auto models. `llm-routing-auto-agy-ollama` chains both: agy first, then Ollama if agy is exhausted, both gated on advanced classification. The `agent-advanced-core` tier itself is a plain LiteLLM tier with no premium trigger. See §2 for the full routing table.*
 
 ### C. Valkey Caching (`redis_settings` in LiteLLM)
@@ -444,6 +509,8 @@ This endpoint outputs plain-text metrics (`Content-Type: text/plain; version=0.0
 | `circuit_breaker_tier` | gauge | Current circuit breaker cooldown tier (0=open, 1-3=cooldown) |
 | `circuit_breaker_agy_allowed` | gauge | Whether agy proxy requests are currently allowed (0/1) |
 | `circuit_breaker_cooldown_remaining_seconds` | gauge | Seconds until cooldown expires |
+| `ollama_cooldown_active` | gauge | Whether Ollama is in router-side cooldown (1=active, 0=open) |
+| `ollama_cooldown_remaining_seconds` | gauge | Seconds remaining in Ollama cooldown |
 | `circuit_breaker_total_trips` | counter | Total number of circuit breaker trips |
 | `circuit_breaker_probe_granted` | gauge | Whether a probe request has been granted (0/1) |
 
@@ -676,29 +743,35 @@ LiteLLM calls `https://api.ollama.com/api/chat` with Bearer authentication using
 
 | Model | Ollama tag | Purpose |
 |-------|-----------|---------|
-| `ollama-deepseek-v4-pro` | `deepseek-v4-pro` | Primary paid tier — 1.6T parameter model |
+| `ollama-deepseek-v4-pro` | `deepseek-v4-pro` | Primary paid tier — 1.6T parameter reasoning & advanced model |
+| `ollama-deepseek-v4-flash` | `deepseek-v4-flash` | Lightweight paid tier — fast complex & below model |
 
 Additional Ollama.com models can be added to `litellm/config.yaml` using the same
 `ollama_chat/` prefix pattern.
 
-### Fallback Chain
+### Fallback and Cooldown Behavior
 
-```
-ollama-deepseek-v4-pro → agent-advanced-core → openrouter-auto
-```
+To prevent cascading fallback loops where a rate-limited Ollama backend repeatedly receives requests from different tiers, the **Triage Router manages Ollama cooldowns internally** rather than relying on LiteLLM's deployment cooldown mechanism (which is unreliable for single-deployment model groups in Community Edition).
+
+The cooldown mechanism works as follows:
 
-If Ollama.com is rate-limited or unavailable, LiteLLM automatically falls back through
-the agent tier cascade to OpenRouter's free model pool.
+1. **Failure Detection**: When calls to `ollama-deepseek-v4-pro` or `ollama-deepseek-v4-flash` fail (due to rate limiting, 429/502/503 errors, or connection issues), the failure is caught by the Triage Router.
+2. **Router-Side Cooldown Activation**: The Triage Router activates an internal **5-minute cooldown** (configurable via `OLLAMA_COOLDOWN_SECONDS` env var). During this window, all subsequent Ollama requests are **immediately rejected** without making any LiteLLM calls.
+3. **Direct / Fallback Requests (`llm-routing-ollama`)**:
+ - During cooldown, the Triage Router returns an HTTP `429` immediately.
+ - LiteLLM receives this 429, skips `llm-routing-ollama` in the fallback chain, and cascades directly to `openrouter-auto`.
+4. **Auto-Routing Requests (`llm-routing-auto-ollama` or `llm-routing-auto-agy-ollama`)**:
+ - During cooldown, the Triage Router silently falls back to the original classified free tier model (e.g., `agent-advanced-core`), querying LiteLLM for a free model.
 
 ### Routing Modes
 
 | Model | Behavior |
 |-------|----------|
-| `llm-routing-ollama` | Direct — skips classifier, goes straight to Ollama deepseek-v4-pro |
-| `llm-routing-auto-ollama` | Auto — classifier runs first, then Ollama deepseek-v4-pro is tried regardless of tier |
-| `llm-routing-auto-agy-ollama` | Chained — classifier runs, tries agy first, then chains to Ollama if agy exhausted |
+| `llm-routing-ollama` | **Gated direct**: runs classifier, routes reasoning & advanced → `ollama-deepseek-v4-pro`, complex & below → `ollama-deepseek-v4-flash` |
+| `llm-routing-auto-ollama` | **Gated auto**: runs classifier, reasoning & advanced → `ollama-deepseek-v4-pro`, complex → `ollama-deepseek-v4-flash`, below → bypasses Ollama to LiteLLM free tiers |
+| `llm-routing-auto-agy-ollama` | **Gated chained**: runs classifier, tries agy first (advanced/reasoning only), then chains to Ollama if agy is exhausted |
 
-All three modes ultimately fall back to LiteLLM's agent tier cascade if the premium backends fail.
+For auto-routing modes, the Triage Router handles failures by silently falling back to the classified free tier cascade. For direct requests to `llm-routing-ollama`, the router returns `429` immediately during cooldown, allowing LiteLLM to skip this model group and cascade to `openrouter-auto`. The cooldown status is visible via the `/metrics` endpoint (`ollama_cooldown_active` and `ollama_cooldown_remaining_seconds` gauges).
 
 ## 9d. Hermes Cron Jobs & Health Check Alerting
 

litellm/config.yaml

@@ -1,5 +1,5 @@
 general_settings:
- master_key: sk-lit...33bf
+ master_key: os.environ/LITELLM_MASTER_KEY
 litellm_settings:
  # -------------------------------------------------------------------------
  # FALLBACK CHAINS (cascading, in order of escalation)
@@ -26,35 +26,44 @@ litellm_settings:
  - agent-complex-core
  - agent-reasoning-core
  - agent-advanced-core
+ - llm-routing-ollama
  - openrouter-auto
  # - local-qwen-3.6 # DISABLED: 35B model unloaded (23GB GTT saved)
  - agent-medium-core:
  - agent-complex-core
  - agent-reasoning-core
  - agent-advanced-core
+ - llm-routing-ollama
  - openrouter-auto
  # - local-qwen-3.6 # DISABLED
  - agent-complex-core:
  - agent-reasoning-core
  - agent-advanced-core
+ - llm-routing-ollama
  - openrouter-auto
  # - local-qwen-3.6 # DISABLED
  - agent-reasoning-core:
  - agent-advanced-core
+ - llm-routing-ollama
  - openrouter-auto
  # - local-qwen-3.6 # DISABLED
  - agent-advanced-core:
+ - llm-routing-ollama
  - openrouter-auto
  # - local-qwen-3.6 # DISABLED
- - ollama-deepseek-v4-pro:
- - agent-advanced-core
- - ollama-deepseek-v4-flash:
- - agent-reasoning-core
+
 model_list:
 - litellm_params:
  model: openrouter/openrouter/auto
  request_timeout: 120
  model_name: openrouter-auto
+- litellm_params:
+ model: openai/llm-routing-ollama
+ api_base: http://127.0.0.1:5000/v1
+ api_key: os.environ/LITELLM_MASTER_KEY
+ request_timeout: 120
+ model_name: llm-routing-ollama
+
 # DISABLED 2026-06-08 — 20GB on disk, 23GB GTT (system RAM as GPU buffer).
 # Uncomment to re-enable once a lighter model replaces it.
 #- litellm_params:
@@ -73,7 +82,10 @@ model_list:
 # ================================================================================
 # OLLAMA PAID TIER — ollama.com via LiteLLM's native ollama_chat provider.
 # LiteLLM calls https://api.ollama.com/api/chat with Bearer auth (OLLAMA_API_KEY).
-# Fallback: ollama-deepseek-v4-pro → agent-advanced-core → openrouter-auto.
+# No LiteLLM-level fallbacks: failures propagate back to the triage router
+# (router/main.py) which manages Ollama cooldowns internally. When Ollama fails,
+# the router activates a 5-minute cooldown and skips Ollama on subsequent
+# requests, returning 429 so LiteLLM falls through to openrouter-auto.
 # ================================================================================
 - model_name: ollama-deepseek-v4-pro
  litellm_params:
@@ -84,7 +96,7 @@ model_list:
 
 # ================================================================================
 # OLLAMA FLASH TIER — lighter/faster model for reasoning-tier requests.
-# Fallback: ollama-deepseek-v4-flash → agent-reasoning-core → openrouter-auto.
+# Same cooldown architecture as the pro tier above.
 # ================================================================================
 - model_name: ollama-deepseek-v4-flash
  litellm_params:
@@ -119,43 +131,38 @@ model_list:
 
 - model_name: agent-advanced-core
  litellm_params:
- model: openrouter/minimax/minimax-m2.5:free
- request_timeout: 120
+ model: openrouter/google/gemma-4-31b-it:free
+ request_timeout: 20
 - model_name: agent-reasoning-core
  litellm_params:
- model: openrouter/moonshotai/kimi-k2.6:free
- request_timeout: 120
+ model: openrouter/google/gemma-4-26b-a4b-it:free
+ request_timeout: 20
 - model_name: agent-complex-core
  litellm_params:
  model: openrouter/nvidia/nemotron-3-ultra-550b-a55b:free
- request_timeout: 120
+ request_timeout: 20
 - model_name: agent-medium-core
  litellm_params:
  model: openrouter/google/gemma-4-26b-a4b-it:free
- request_timeout: 120
+ request_timeout: 20
 - model_name: agent-simple-core
  litellm_params:
  model: openrouter/nvidia/nemotron-3-nano-30b-a3b:free
- request_timeout: 120
+ request_timeout: 20
 
 redis_settings:
  redis_host: 127.0.0.1
  redis_port: 6379
 router_settings:
- allowed_fails: 2
+ allowed_fails: 0
  cooldown_time: 300
  enable_pre_call_checks: false
  num_retries: 1
  routing_strategy: latency-based-routing
- # Per-error-type cooldown thresholds (overrides allowed_fails for specific errors).
- # Upstream rate limits ("temporarily rate-limited upstream") can last minutes —
- # a 30s cooldown just wastes retries. 300s gives providers time to clear the limit.
- # RateLimitError: cooldown on FIRST 429 (upstream rate limits persist for minutes —
- # allowing even 1 retry wastes a request against a provider that's still throttling)
- allowed_fails_policy:
- RateLimitErrorAllowedFails: 0
- TimeoutErrorAllowedFails: 3
- BadRequestErrorAllowedFails: 1
+ enable_health_check_routing: false
+ # NOTE: allowed_fails_policy is an enterprise-only feature in LiteLLM.
+ # Ollama cooldowns are handled by the triage router itself (router/main.py),
+ # not by LiteLLM's deployment cooldown mechanism.
 vector_store_settings:
  collection_name: litellm_semantic_cache
  connection_string: postgresql://postgres:***@127.0.0.1:5432/postgres