RuntimeRouter

Optimize the entire AI session, not just the next model call.

RuntimeRouter is a Python-first LLM routing library for the AI ecosystem. It sits above agent frameworks — LangGraph, PydanticAI, CrewAI, AutoGen — as a Runtime Router, not a replacement for them.

You write agents with your favorite framework. RuntimeRouter decides which model to call, when, and why — across the full session lifecycle.

---

Vision

Traditional routers optimize a single inference call: given a prompt, pick the cheapest or fastest model.

RuntimeRouter's long-term goal is different: optimize the entire AI session — model selection, cost, latency, context, caching, and privacy — as a unified runtime decision layer.

┌─────────────────────────────────────────────────────────┐
│ Your Agent Framework (LangGraph / PydanticAI / …) │
├─────────────────────────────────────────────────────────┤
│ RuntimeRouter ← session-aware routing (this library) │
├─────────────────────────────────────────────────────────┤
│ LiteLLM / Provider APIs (OpenAI, Anthropic, …) │
└─────────────────────────────────────────────────────────┘

---

Why RuntimeRouter?

Problem	RuntimeRouter approach
Hard-coding model="gpt-4o" everywhere	AutoLLM() picks the right model per request
Simple tasks burning frontier-model budget	Complexity routing sends easy tasks to cheap models
No visibility into routing decisions	Every call returns a RouteDecision with reason
Framework lock-in	Framework-agnostic; works with any LiteLLM-compatible stack
Proxy-only routers	Designed for session-level optimization (roadmap)

---

vs OpenRouter

	OpenRouter	RuntimeRouter
What it is	Unified API proxy to 100+ models	Python routing library embedded in your app
Scope	Single HTTP request → model	Entire AI session (roadmap)
Integration	Replace your API base URL	Drop-in AutoLLM() or Router in Python code
Policies	Provider-side routing rules	Pluggable Python Policy classes you own
Framework	Language-agnostic HTTP	Python-first, native LangGraph/PydanticAI hooks (v0.2+)

OpenRouter is excellent as a model gateway. RuntimeRouter is a runtime decision engine you control in-process.

---

vs Not Diamond

	Not Diamond	RuntimeRouter
What it is	Managed routing SaaS	Open-source Python library
Deployment	Cloud API	In-process, self-hosted
Customization	Platform-configured	Full plugin Policy architecture
Session scope	Per-call model selection	Session-aware optimization (roadmap)
Cost	SaaS pricing	Free & open (MIT)

Not Diamond provides intelligent routing as a service. RuntimeRouter gives you the same concept as extensible, auditable Python code.

---

Quick Start

Install

pip install runtimerouter

Basic usage

fromruntimerouterimportAutoLLMllm=AutoLLM()
# RuntimeRouter automatically selects Claude, Gemini, DeepSeek, OpenAI, etc.response=llm.invoke("帮我分析整个代码仓库")
print(response.choices[0].message.content)

Inspect routing decisions

decision=llm.route_only("Summarize this paragraph in one sentence.")
print(decision.selected_model) # e.g. "gpt-4o-mini"print(decision.reason) # why this model was chosenprint(decision.complexity) # e.g. ComplexityLevel.SIMPLE

Configure policies

fromruntimerouterimportAutoLLM, RouterConfigconfig=RouterConfig(
enable_complexity_routing=True,
enable_cost_routing=True,
fallback_model="gpt-4o-mini",
policy_order=["complexity", "cost"],
)
llm=AutoLLM(config=config)

Use Router directly (without invoking)

fromruntimerouterimportRouter, RouteContextrouter=Router()
decision=router.route(RouteContext(prompt="Explain quantum entanglement"))
print(decision.selected_model)

---

Architecture

runtimerouter/
├── autollm.py # User-facing entry point
├── router.py # Routing pipeline orchestrator
├── classifier.py # Task complexity & token estimation
├── types.py # Shared data contracts (Pydantic models)
├── config.py # Config loading utilities
├── policies/ # Pluggable routing policies
│ ├── base.py # RoutingPolicy ABC
│ ├── complexity.py # Complexity-based routing
│ └── cost.py # Cost-based routing
├── providers/ # Model catalog & provider metadata
│ ├── base.py # ModelProvider ABC
│ └── registry.py # ModelRegistry (default model catalog)
└── integrations/ # External execution layers
└── litellm.py # LiteLLM completion wrapper

Routing pipeline (v0.1):

User prompt
│
▼
TaskClassifier.enrich() ← complexity, token estimate
│
▼
Policy chain (ordered) ← complexity → cost
│
▼
RouteDecision ← selected model + reason
│
▼
LiteLLMIntegration ← execute completion

See docs/architecture.md for module boundaries and extension points.

---

Roadmap

Version	Focus
v0.1(current)	Auto Model Selection, Complexity Routing, Cost Routing, LiteLLM Integration
v0.2	LangGraph Integration, PydanticAI Integration
v0.3	Cache-aware Routing
v0.4	Context-aware Routing
v0.5	Session-aware Optimization

Full details: ROADMAP.md

---

Contributing

We welcome contributions! See CONTRIBUTING.md.

git clone https://github.com/chenyu-dev25/RuntimeRouter.git
cd RuntimeRouter
python -m venv .venv &&source .venv/bin/activate
pip install -e ".[dev]"
pytest

---

License

MIT — see LICENSE.

---

RuntimeRouter
Optimize the entire AI session, not just the next model call.