feat(litellm): add experimental Switchyard integration

[dnandakumar-nv]

What

Add an experimental, source-installable switchyard-litellm package under
examples/experimental/litellm that demonstrates how to combine Switchyard
routing with a Dockerized LiteLLM gateway backed by OpenRouter.

• Add LiteLLMSyClient, a buffered text-only adapter from normalized libsy
  dictionaries to LiteLLM's asynchronous Completion API (acompletion).
• Route with the existing switchyard.libsy.algorithms.random binding between
  openai/gpt-5.6-sol (strong) and moonshotai/kimi-k3 (fast).
• Keep provider model IDs and the OPENROUTER_API_KEY behind LiteLLM gateway
  aliases so application code only selects strong or fast.
• Add a loopback-only Docker Compose setup pinned to
  ghcr.io/berriai/litellm:v1.92.0 and litellm==1.92.0.
• Add offline coverage, an explicit opt-in paid E2E that calls both real
  OpenRouter models, and a three-task Harbor smoke benchmark configuration.
• Add a developer-focused integration guide covering uv installation, gateway
  startup, application usage, testing, benchmarking, limitations, and security.

Why

LiteLLM provides the OpenAI-compatible gateway, provider integration, model
aliases, and credential boundary. Switchyard owns the routing decision. This
example connects the two with installable code and copyable configuration so a
developer can add Switchyard routing without embedding provider-specific model
IDs or credentials in application code.

application → libsy random router → LiteLLMSyClient
            → LiteLLM gateway alias → OpenRouter Chat Completions
            → GPT-5.6 Sol or Kimi K3

How tested

• uv run ruff check . clean
• uv run mypy switchyard clean — 150 source files
• Hermetic uv run pytest tests/ -v green — 1,723 passed, 35 skipped
• Package offline suite — 29 passed, 1 paid E2E deselected
• Wheel/sdist build and clean source-install/import smoke
• Explicit gated paid E2E through Dockerized LiteLLM and OpenRouter — both
  strong and fast aliases passed; isolated Compose teardown verified
• Harbor mini-benchmark — 3/3 tasks completed with reward 1.0; 12 routed
  requests, 0 errors, strong=7, fast=5
• Strict MkDocs build, Compose config validation, SPDX gate, diff check,
  and secret-pattern audit

Checklist

• One class per file; filename matches the primary class.
• Public symbol exported from the nested package's
  switchyard_litellm.__init__; no root Switchyard API change is intended.
• Unit and integration tests added.
• README is the canonical experimental integration guide and is linked from
  the random-routing documentation.
• All commits include DCO sign-off.

Notes for reviewers

• This integration is explicitly experimental. Its README keeps the warning at
  the top and documents the supported request subset and security boundary.
• Paid tests require both OPENROUTER_API_KEY and
  SWITCHYARD_LITELLM_E2E=1; merely having a key exported does not spend.
• The client calls LiteLLM's acompletion with retries disabled so retry and
  routing policy remain with the outer Switchyard layer.
• The example intentionally supports buffered, text-only Chat Completions;
  tools, media, structured output, and streaming fail explicitly.
• The retained Harbor workflow documents a temporary fallback for a host/task
  Codex model-catalog schema mismatch without changing benchmark source.

Summary by CodeRabbit

• New Features

  • Added an experimental LiteLLM integration example with weighted random routing between configured model aliases.
  • Added a Dockerized local gateway setup supporting OpenRouter-backed models.
  • Added an executable example for sending normalized requests through the routing workflow.

• Documentation

  • Added setup, configuration, troubleshooting, security, compatibility, and usage guidance.
  • Documented supported request types and current limitations, including text-only, non-streaming responses.

• Tests

  • Added offline validation and opt-in end-to-end coverage for gateway configuration, request handling, and routing behavior.

[coderabbitai]

Walkthrough

Changes

Adds an experimental LiteLLM integration that routes normalized Switchyard requests through a Dockerized LiteLLM gateway to OpenRouter-backed aliases. It includes the client adapter, weighted routing examples, benchmark configuration, documentation, and offline or opt-in Docker E2E tests.

LiteLLM integration

Layer / File(s)	Summary
Gateway package and configuration examples/experimental/litellm/pyproject.toml, examples/experimental/litellm/src/switchyard_litellm/__init__.py, examples/experimental/litellm/litellm-config.yaml, examples/experimental/litellm/compose.yaml, examples/experimental/litellm/.env.example, examples/experimental/litellm/benchmark-route.yaml, examples/experimental/litellm/benchmark-tasks.txt	Defines the package, LiteLLM aliases, Docker service, OpenRouter credential wiring, and benchmark inputs.
Normalized request and response adapter examples/experimental/litellm/src/switchyard_litellm/client.py	Validates supported normalized requests, calls LiteLLM asynchronously, and converts responses and usage into Switchyard-compatible mappings.
Weighted routing examples and benchmark wiring examples/experimental/litellm/example.py	Creates strong and fast clients, applies deterministic weighted random routing, prints decisions and responses, and closes clients.
Offline adapter and routing validation examples/experimental/litellm/tests/conftest.py, examples/experimental/litellm/tests/test_client.py, examples/experimental/litellm/tests/test_gateway_config.py, examples/experimental/litellm/tests/test_random_routing.py	Tests translation, normalization, validation errors, LiteLLM errors, configuration wiring, token usage, and deterministic target selection.
Opt-in Docker E2E validation examples/experimental/litellm/tests/test_e2e.py	Starts and tears down the Compose gateway only after explicit opt-in, then verifies routing to both configured models.
Integration and benchmark documentation docs/routing_algorithms/random_routing.md, examples/experimental/litellm/README.md	Documents setup, request flow, routing, limitations, testing, benchmarking, troubleshooting, shutdown, security, and references.

Estimated code review effort: 4 (Complex) | ~45 minutes

Poem

I’m a rabbit routing dreams through the gate,
Strong or fast, the seed decides their fate.
Docker hums and aliases bloom,
Tests hop neatly around the room.
OpenRouter stars shine bright—
LiteLLM carries them tonight.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: an experimental LiteLLM-based Switchyard integration.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (2)

examples/experimental/litellm/tests/test_e2e.py (1)

25-26: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Document the public fixture.

Add a concise docstring describing its opt-in behavior, yielded URL, skips, and Compose failure behavior.

Proposed fix

 `@pytest.fixture`(scope="session")
 def litellm_base_url() -> Iterator[str]:
+    """Yield a loopback LiteLLM URL for opt-in paid E2E tests.
+
+    Skips without the opt-in, OpenRouter key, or Docker; raises on Compose failures.
+    """
     if os.environ.get("SWITCHYARD_LITELLM_E2E") != "1":

As per coding guidelines, “Add concise triple-quoted docstrings for public functions, classes, methods, and API entry points.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@examples/experimental/litellm/tests/test_e2e.py` around lines 25 - 26,
Document the public litellm_base_url fixture with a concise triple-quoted
docstring covering its opt-in behavior, the URL it yields, when tests are
skipped, and how Docker Compose failures are handled.

Source: Coding guidelines

examples/experimental/litellm/tests/test_client.py (1)

135-191: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add a tool_choice rejection test case.

The parametrize list covers most _payload validation branches but omits tool_choice, which _payload explicitly rejects when non-None (per client.py's if request.get("tool_choice") is not None: raise ValueError("tool_choice is not supported")). Add a case like (lambda body: body.update(tool_choice="auto"), "tool_choice") to guard this branch against regression.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@examples/experimental/litellm/tests/test_client.py` around lines 135 - 191,
Add a parameterized rejection case to the test list covering `_payload`’s
unsupported tool_choice branch: mutate the body with a non-None tool_choice
value such as "auto" and expect the validation match to identify "tool_choice".
Keep the existing validation cases unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@examples/experimental/litellm/pyproject.toml`:
- Around line 27-28: Add a py.typed marker file under the src/switchyard_litellm
package so installed consumers recognize its inline type information. Ensure the
marker is included in the setuptools package data produced by the existing
package discovery configuration.

In `@examples/experimental/litellm/tests/test_e2e.py`:
- Around line 49-61: Bound both Docker Compose lifecycle calls in the E2E setup
around the startup and teardown subprocess.run invocations: add a finite
--wait-timeout argument to the compose up command and timeout= values to both
subprocess.run calls, ensuring startup and cleanup cannot hang indefinitely.

---

Nitpick comments:
In `@examples/experimental/litellm/tests/test_client.py`:
- Around line 135-191: Add a parameterized rejection case to the test list
covering `_payload`’s unsupported tool_choice branch: mutate the body with a
non-None tool_choice value such as "auto" and expect the validation match to
identify "tool_choice". Keep the existing validation cases unchanged.

In `@examples/experimental/litellm/tests/test_e2e.py`:
- Around line 25-26: Document the public litellm_base_url fixture with a concise
triple-quoted docstring covering its opt-in behavior, the URL it yields, when
tests are skipped, and how Docker Compose failures are handled.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c6ae4b14-8a1b-4bae-8f6c-6f075ec5166a

📥 Commits

Reviewing files that changed from the base of the PR and between 11d3648 and ab3678a.

⛔ Files ignored due to path filters (1)

• examples/experimental/litellm/uv.lock is excluded by !**/*.lock

📒 Files selected for processing (16)

• docs/routing_algorithms/random_routing.md
• examples/experimental/litellm/.env.example
• examples/experimental/litellm/README.md
• examples/experimental/litellm/benchmark-route.yaml
• examples/experimental/litellm/benchmark-tasks.txt
• examples/experimental/litellm/compose.yaml
• examples/experimental/litellm/example.py
• examples/experimental/litellm/litellm-config.yaml
• examples/experimental/litellm/pyproject.toml
• examples/experimental/litellm/src/switchyard_litellm/__init__.py
• examples/experimental/litellm/src/switchyard_litellm/client.py
• examples/experimental/litellm/tests/conftest.py
• examples/experimental/litellm/tests/test_client.py
• examples/experimental/litellm/tests/test_e2e.py
• examples/experimental/litellm/tests/test_gateway_config.py
• examples/experimental/litellm/tests/test_random_routing.py

[nachiketb-nvidia]

lgtm for the example and first pass, we'll come back to this once we have changes to benchmarking and binding for better algos