feat(router): add OpenAI Responses API and Home Assistant model support (#374) - #382
feat(router): add OpenAI Responses API and Home Assistant model support (#374)#382sheepdestroyer wants to merge 4 commits into
Conversation
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
Reviewer's GuideAdds an OpenAI Responses API proxy endpoint with auto-routing and streaming support, introduces Home Assistant-friendly model aliases and visibility, and documents and tests the new Responses behavior and tools integration. Sequence diagram for OpenAI Responses API proxy with auto-routing and streamingsequenceDiagram
actor HomeAssistant
participant Router as responses_api
participant Classifier as classify_request
participant LiteLLM as LiteLLM_v1_responses
HomeAssistant->>Router: POST /v1/responses (model or auto model)
Router->>Router: sync_cooldowns_from_valkey
Router->>Router: parse last_user_message from input/instructions
alt auto model requested
Router->>Classifier: classify_request(last_user_message, bypass_cache)
Classifier-->>Router: target_model
else direct model alias
Router->>Router: target_model = client_model
end
Router->>Router: body_to_send[model] = target_model
Router->>Router: get_http_client
alt stream == true
Router->>LiteLLM: client.stream("POST", /v1/responses, json=body_to_send)
LiteLLM-->>Router: SSE chunks
Router-->>HomeAssistant: StreamingResponse(text/event-stream)
else non-streaming
Router->>LiteLLM: client.post(/v1/responses, json=body_to_send)
LiteLLM-->>Router: JSON response
Router-->>HomeAssistant: Response(status_code, headers, content)
end
File-Level Changes
Assessment against linked issues
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Warning Review limit reached
Next review available in:53 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughAdds OpenAI Responses API and audio proxy endpoints, Home Assistant-compatible model aliases, direct routing updates, transcription URL fixes, automated coverage, and README documentation for configuration and compatibility. ChangesResponses API and Home Assistant integration
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant Router
participant Classifier
participant LiteLLM
Client->>Router: POST /v1/responses
Router->>Classifier: Classify auto-routed request
Classifier-->>Router: Selected target model
Router->>LiteLLM: Proxy Responses API request
LiteLLM-->>Router: JSON response or SSE stream
Router-->>Client: Forward response and function-call events
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Hey - I've found 1 issue, and left some high level feedback:
- The triage and proxy logic in
responses_apilargely duplicates the behavior inchat_completions; consider extracting a shared helper for model selection, auth header construction, and proxying to LiteLLM to keep these endpoints consistent as behavior evolves. - The
AUTO_MODELSset is now defined separately in bothresponses_apiandchat_completions; centralizing this configuration (e.g., as a module-level constant) would reduce the chance of model list drift between endpoints.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments- The triage and proxy logic in `responses_api` largely duplicates the behavior in `chat_completions`; consider extracting a shared helper for model selection, auth header construction, and proxying to LiteLLM to keep these endpoints consistent as behavior evolves.
- The `AUTO_MODELS` set is now defined separately in both `responses_api` and `chat_completions`; centralizing this configuration (e.g., as a module-level constant) would reduce the chance of model list drift between endpoints.
## Individual Comments### Comment 1
<locationpath="router/main.py"line_range="1999-2024" />
<code_context>
+ input_field = body.get("input")
+ if isinstance(input_field, str):
+ last_user_message = input_field+ elif isinstance(input_field, list):
+ for item in input_field:+ if isinstance(item, str):+ last_user_message += item+ elif isinstance(item, dict):+ if item.get("type") == "text":+ last_user_message += item.get("text", "")+ elif item.get("role") == "user":+ content = item.get("content") or ""+ if isinstance(content, str):+ last_user_message += content+ elif isinstance(content, list):+ last_user_message += "".join(+ b.get("text", "")+ for b in content+ if isinstance(b, dict) and b.get("type") == "text"+ )++ if not last_user_message:
</code_context>
<issue_to_address>
**suggestion:** Concatenating multiple input segments without separators may degrade triage classification quality.
When `input` is a list, all segments are concatenated into `last_user_message` without any separators. For long or structured inputs this can obscure boundaries between messages or content types and negatively impact `classify_request`. Consider inserting simple delimiters (spaces/newlines) between segments or restricting which parts are included in triage to improve routing accuracy.
```suggestion last_user_message = "" input_field = body.get("input") if isinstance(input_field, str): last_user_message = input_field elif isinstance(input_field, list): # Collect segments and join with a delimiter to preserve boundaries for triage segments: list[str] = [] for item in input_field: if isinstance(item, str): segments.append(item) elif isinstance(item, dict): if item.get("type") == "text": segments.append(item.get("text", "")) elif item.get("role") == "user": content = item.get("content") or "" if isinstance(content, str): segments.append(content) elif isinstance(content, list): segments.append( "".join( b.get("text", "") for b in content if isinstance(b, dict) and b.get("type") == "text" ) ) # Use a simple space delimiter to avoid degrading classification by merging segments last_user_message = " ".join(s for s in segments if s) if not last_user_message: instructions = body.get("instructions") if isinstance(instructions, str): last_user_message = instructions```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
litellm/config.yaml (1)
124-165: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winThree near-identical
litellm_params/model_infoblocks — consider YAML anchors.
local-qwen-3.6-hass,gpt-4o-mini, andgpt-4orepeat byte-identicallitellm_paramsandmodel_infoblocks. YAML anchors/merge keys (&anchor/<<: *anchor) would remove the duplication and reduce drift risk if one copy is updated but the others are forgotten (as likely happened with thesupports_reasoningflag above).♻️ Example using YAML anchors
- litellm_params: &hass_paramsapi_base: LLAMA_CLASSIFIER_URL_PLACEHOLDERapi_key: local-tokenmodel: openai/local-qwen-3.6-hassextra_body: chat_template_kwargs: enable_thinking: falserequest_timeout: 600model_name: local-qwen-3.6-hassmodel_info: &hass_model_infosupports_vision: truesupports_reasoning: falsesupports_function_calling: truemode: chatmax_tokens: 524288max_input_tokens: 524288is_public_model_group: true - litellm_params: *hass_paramsmodel_name: gpt-4o-minimodel_info: *hass_model_info - litellm_params: *hass_paramsmodel_name: gpt-4omodel_info: *hass_model_info🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@litellm/config.yaml` around lines 124 - 165, Replace the duplicated litellm_params and model_info mappings for local-qwen-3.6-hass, gpt-4o-mini, and gpt-4o with YAML anchors and aliases. Define each shared mapping once, then reference it from the three model entries while preserving their distinct model_name values and the intended supports_reasoning setting.router/tests/test_responses_api.py (1)
1-177: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winMissing coverage for structured
inputarrays and streaming error paths.All five tests use a plain string
input; none exercise the list-basedinputshape ([{"role": "user", "content": [{"type": "input_text", ...}]}]) that real Responses API clients (including the OpenAI SDK) send for multi-part/multi-turn requests. That gap is exactly why thetype == "text"vsinput_textbug inresponses_api's extraction logic (seerouter/main.pyreview) went untested. Likewise,test_responses_api_streamingonly simulates a successful stream — there's no test asserting behavior when the upstream stream returns a non-200 status, which is why the missing status-code check in the streaming branch wasn't caught.Consider adding:
- A test with
"input": [{"role": "user", "content": [{"type": "input_text", "text": "..."}]}]asserting the classifier receives the extracted text.- A test where
mock_client.stream/upstream returns a non-200 status during a streaming request, asserting an appropriate error is surfaced rather than a 200.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@router/tests/test_responses_api.py` around lines 1 - 177, Extend the response API tests with structured input coverage: add a list-based input containing input_text content and assert responses_api passes the extracted text to classify_request. Add a streaming failure test using a non-200 upstream response from mock_client.stream, asserting responses_api surfaces an appropriate error response instead of returning a successful StreamingResponse.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@litellm/config.yaml`:
- Around line 124-165: Update the model_info entries for the
local-qwen-3.6-hass, gpt-4o-mini, and gpt-4o aliases to set supports_reasoning
to false. Leave the existing enable_thinking configuration and other capability
flags unchanged, matching the local-qwen-3.6 behavior.
In `@router/main.py`:
- Around line 2050-2072: Update the streaming branch in responses_api’s
response_streamer to validate the upstream resp.status_code before yielding any
bytes. Propagate non-success responses as HTTP errors, preserving the existing
successful StreamingResponse behavior and matching the status handling used by
chat_completions.
- Around line 1999-2020: Update the last-user-input extraction in responses_api
to recognize input_text for both top-level items and nested content blocks, and
change the input-list traversal to inspect items in reverse and stop after
extracting the latest user turn. Keep chat_completions aligned with this
behavior by selecting only the most recent user message rather than
concatenating multiple turns.
---
Nitpick comments:
In `@litellm/config.yaml`:
- Around line 124-165: Replace the duplicated litellm_params and model_info
mappings for local-qwen-3.6-hass, gpt-4o-mini, and gpt-4o with YAML anchors and
aliases. Define each shared mapping once, then reference it from the three model
entries while preserving their distinct model_name values and the intended
supports_reasoning setting.
In `@router/tests/test_responses_api.py`:
- Around line 1-177: Extend the response API tests with structured input
coverage: add a list-based input containing input_text content and assert
responses_api passes the extracted text to classify_request. Add a streaming
failure test using a non-200 upstream response from mock_client.stream,
asserting responses_api surfaces an appropriate error response instead of
returning a successful StreamingResponse.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 7e431313-2d14-4759-a6b2-c78600166af2
📒 Files selected for processing (4)
README.mdlitellm/config.yamlrouter/main.pyrouter/tests/test_responses_api.py
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…one event injection for HA Assist
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@router/main.py`:
- Around line 2144-2145: Update the streaming loops around resp.aiter_bytes() to
preserve raw bytes across transport chunks instead of decoding each chunk with
errors="replace"; accumulate byte data and decode only after complete SSE
boundaries are identified, including the analogous path noted near the second
occurrence.
- Around line 2170-2188: Update the synthesized delta and done events in the
response streaming handler to derive output_index and sequence_number from the
corresponding upstream event or item metadata instead of hardcoding both to
zero. Preserve the existing item_id, function name, arguments, and deduplication
behavior while ensuring each function call retains its upstream output position
and event ordering.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: e4834f5a-2937-4d30-af1c-2fef409344f0
📒 Files selected for processing (2)
litellm/config.yamlrouter/main.py
🚧 Files skipped from review as they are similar to previous changes (1)
- litellm/config.yaml
| async for chunk in resp.aiter_bytes(): | ||
| buffer += chunk.decode("utf-8", errors="replace") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Preserve raw SSE bytes across transport chunks.
HTTP chunks need not align to UTF-8 boundaries; decoding each chunk with errors="replace" can corrupt streamed text or function arguments before they are forwarded.
Proposed fix
- buffer = ""+ buffer = b""
seen_args_delta = set()
seen_args_done = set()
async for chunk in resp.aiter_bytes():
- buffer += chunk.decode("utf-8", errors="replace")- while "\n" in buffer:- line, buffer = buffer.split("\n", 1)+ buffer += chunk+ while b"\n" in buffer:+ line, buffer = buffer.split(b"\n", 1)
line_str = line.strip()
- if line_str.startswith("data:"):+ if line_str.startswith(b"data:"):
raw_data = line_str[5:].strip()
...
- yield (line + "\n").encode("utf-8")+ yield line + b"\n"
if buffer:
- yield buffer.encode("utf-8")+ yield bufferAlso applies to: 2191-2193
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@router/main.py` around lines 2144 - 2145, Update the streaming loops around
resp.aiter_bytes() to preserve raw bytes across transport chunks instead of
decoding each chunk with errors="replace"; accumulate byte data and decode only
after complete SSE boundaries are identified, including the analogous path noted
near the second occurrence.
| delta_evt = { | ||
| "type": "response.function_call_arguments.delta", | ||
| "item_id": item_id, | ||
| "delta": args_val, | ||
| "output_index": 0, | ||
| "sequence_number": 0, | ||
| } | ||
| yield f"data: {json.dumps(delta_evt)}\n\n".encode("utf-8") | ||
| if item_id and item_id not in seen_args_done: | ||
| seen_args_done.add(item_id) | ||
| done_evt = { | ||
| "type": "response.function_call_arguments.done", | ||
| "item_id": item_id, | ||
| "name": item.get("name", ""), | ||
| "arguments": args_val, | ||
| "output_index": 0, | ||
| "sequence_number": 0, | ||
| } | ||
| yield f"data: {json.dumps(done_evt)}\n\n".encode("utf-8") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bashset -euo pipefail
rg -n -C 5 \
'response\.output_item\.done|function_call_arguments\.(delta|done)|output_index|sequence_number' \
router/main.py router/tests/test_responses_api.pyRepository: sheepdestroyer/LLM-Routing
Length of output: 4308
🏁 Script executed:
#!/bin/bashset -euo pipefail
echo"== router/main.py relevant context =="
sed -n '2080,2210p' router/main.py
echoecho"== search for output_index/sequence_number in repo =="
rg -n 'output_index|sequence_number|response\.output_item\.done|response\.function_call_arguments\.(delta|done)' router ||trueRepository: sheepdestroyer/LLM-Routing
Length of output: 8009
🌐 Web query:
OpenAI Responses streaming event function_call_arguments.delta output_index sequence_number
💡 Result:
When streaming function call arguments in OpenAI's API (specifically within the context of the Responses API or Assistants API), the response.function_call_arguments.delta event is used to stream the incremental parts of the JSON arguments string [1][2][3]. The event object typically includes the following key fields: - delta: A string containing the next portion of the function arguments [2][4]. This should be appended to your existing buffer to reconstruct the full JSON string [2][3]. - item_id: The ID of the specific output item (function call) this delta belongs to [2][4]. - output_index: The integer index of the output item within the response, which helps track multiple concurrent or sequential function calls [2][3][4]. - sequence_number: An integer representing the sequence order of this specific event [1][3][4]. In practice, you should listen for these events, use the output_index (or item_id) to route the delta to the correct function call buffer, and append the delta string to that buffer [2][3]. Do not attempt to parse the arguments as JSON until you receive the corresponding response.function_call_arguments.done event, which signifies that the full argument string has been received [2][3]. Note that this streaming pattern differs from the standard Chat Completions API streaming (which uses tool_calls deltas within chunks) [5][6]; the event-based structure described above is specific to the Responses/Assistants API streaming architecture [7][3].
Citations:
- 1: https://developers.openai.com/api/reference/resources/responses/streaming-events/
- 2: https://developers.openai.com/api/docs/guides/function-calling
- 3: https://community.openai.com/t/responses-api-streaming-the-simple-guide-to-events/1363122
- 4: https://github.com/openai/openai-python/blob/main/src/openai/types/responses/response_function_call_arguments_delta_event.py
- 5: https://github.com/openai/openai-python/blob/5e8f09c2/src/openai/types/chat/chat_completion_chunk.py
- 6: https://theneuralbase.com/function-calling/learn/intermediate/detecting-tool-call-start-in-stream/
- 7: https://developers.openai.com/api/docs/guides/streaming-responses
🌐 Web query:
site:platform.openai.com/docs/api-reference/responses-streaming response.output_item.done output_index sequence_number
💡 Result:
response.output_item.done includes:
output_index: the index of the output item in the response’soutputarray.sequence_number: the sequence number of the event, used to order streaming events. It’s an integer.type: alwaysresponse.output_item.done. (platform.openai.com)
Example:
{
"type": "response.output_item.done",
"output_index": 0,
"item": { "...": "..." },
"sequence_number": 1
}
``` ([platform.openai.com](https://platform.openai.com/docs/api-reference/responses-streaming/response/incomplete?lang=node.js&utm_source=openai))Citations:- 1: https://platform.openai.com/docs/api-reference/responses-streaming/response/incomplete?lang=node.js&utm_source=openai- 2: https://platform.openai.com/docs/api-reference/responses-streaming/response/incomplete?lang=node.js&utm_source=openai---🌐 Web query:`site:platform.openai.com/docs/api-reference/responses-streaming response.function_call_arguments.delta "output_index"`💡 Result:`response.function_call_arguments.delta` includes `output_index`, which is **the index of the output item that the function-call arguments delta is added to**. It’s an integer field in the streaming event payload. ([platform.openai.com](https://platform.openai.com/docs/api-reference/responses-streaming/response/refusal/delta?lang=curl&utm_source=openai))Citations:- 1: https://platform.openai.com/docs/api-reference/responses-streaming/response/refusal/delta?lang=curl&utm_source=openai---</details>**Derive synthesized function-call event metadata from upstream events.**These injected `response.function_call_arguments.delta` / `.done` events always use `output_index: 0` and `sequence_number: 0`, but `output_index` must reflect the output item the function call belongs to, and `sequence_number` is used to order streaming events. For multi-function-call responses, this can mislabel tool streams and create event-ordering conflicts.<details><summary>🧰 Tools</summary><details><summary>🪛 ast-grep (0.44.1)</summary>
[info] 2176-2176: use jsonify instead of json.dumps for JSON outputContext: json.dumps(delta_evt)Note: [CWE-116] Improper Encoding or Escaping of Output.(use-jsonify)---
[info] 2187-2187: use jsonify instead of json.dumps for JSON outputContext: json.dumps(done_evt)Note: [CWE-116] Improper Encoding or Escaping of Output.(use-jsonify)</details></details><details><summary>🤖 Prompt for AI Agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @router/main.py around lines 2170 - 2188, Update the synthesized delta and
done events in the response streaming handler to derive output_index and
sequence_number from the corresponding upstream event or item metadata instead
of hardcoding both to zero. Preserve the existing item_id, function name,
arguments, and deduplication behavior while ensuring each function call retains
its upstream output position and event ordering.
</details>
<!-- fingerprinting:phantom:poseidon:terra -->
<!-- cr-indicator-types:potential_issue -->
<!-- cr-comment:v1:b16ce255483158d45a31475c -->
<!-- This is an auto-generated comment by CodeRabbit -->
- Set supports_reasoning to false in litellm/config.yaml for non-thinking hass aliases (local-qwen-3.6-hass, gpt-4o-mini, gpt-4o) - Support input_text and text content parts, traverse input in reverse to extract the latest user turn for triage, and join multi-part text with space separators - Validate upstream status_code in streaming branch before returning StreamingResponse and close stream on exit - Expand pytest suite to cover input_text parsing and streaming error status propagation
- Parameterize test_responses_api_with_tools and test_responses_api_streaming_tool_calls across local-qwen-3.6-hass, gpt-4o-mini, and gpt-4o - Add test_ha_tool_calling function to scripts/verification/verify_canonical_endpoints.py to replicate Home Assistant Assist tool call execution via /v1/responses and /v1/chat/completions
sheepdestroyer
commented
Jul 26, 2026
Closing in favor of a clean, updated PR incorporating all review feedback and tool-calling E2E validation. |
Summary
This PR addresses issue #374 by implementing OpenAI Responses API support and model aliases required by Home Assistant's
openai_conversationintegration.Changes Included
Responses API Endpoints:
POST /v1/responsesandPOST /responseshandlers inrouter/main.pythat proxy requests to LiteLLM's/v1/responsesendpoint.llm-routing-auto-free) are requested.Model Aliases:
gpt-4o-miniandgpt-4omodel definitions tolitellm/config.yamlrouting tolocal-qwen-3.6-hass(thinking disabled).local-qwen-3.6,local-qwen-3.6-hass,gpt-4o-mini, andgpt-4otopublic_model_groupssoGET /v1/modelslists them.DIRECT_TIERSinrouter/main.pyto bypass classifier overhead when specified directly.Tool Support:
functiontool type returningfunction_calloutput),code_interpreter, andweb_search.Testing & Documentation:
router/tests/test_responses_api.pycovering model routing, Responses API requests, tools, streaming, and error handling (363/363 tests passing).README.mddetailing Home Assistant configuration and capability requirements.wiki/entities/llm-routing.mdandwiki/log.md).Closes#374
Summary by Sourcery
Add OpenAI Responses API proxy support and Home Assistant-oriented model aliases and documentation, including tests for the new responses endpoint and routing behavior.
New Features:
Enhancements:
Tests:
Summary by CodeRabbit
POST /v1/responsesandPOST /responses, including auto model routing and streaming support./v1/audio{path}and/audio{path}.502on upstream proxy failures.