Uh oh!
There was an error while loading. Please reload this page.
feat(responses): emit UsageEvent on /v1/responses 200 non-streaming (#404 MVP) - #425
Merged
Merged
Conversation
…404) Pre-#404, /v1/responses dropped the UsageEvent entirely. Every o1/o3/GPT-5 traffic through OpenAI's Responses API (the modern default entry point for reasoning models) was invisible to cp-api's budget ledger and customer-facing /logs analytics. This PR ships the non-streaming MVP. The handler now extracts the upstream `usage` block and emits a UsageEvent with: - `prompt_tokens` = usage.input_tokens - `completion_tokens` = usage.output_tokens - `reasoning_tokens` = usage.output_tokens_details.reasoning_tokens (uniquely surfaced for o1/o3/GPT-5 class models) - `cached_prompt_tokens` = usage.input_tokens_details.cached_tokens (OpenAI prompt-cache hit subset of input_tokens) - `status_code`, `model_id`, `api_key_id`, `latency_ms` - `inbound_protocol` = "openai" Streaming path scope-deferred: SSE byte-stream interception is needed to extract usage from the final `response.completed` event, and the current responses.rs handler passes the byte stream through verbatim. Tracked as a #404 streaming follow-up — same MVP pattern #402 used for embeddings. Emit-on-success-only: - non-stream 200 with `usage` present → emit - non-stream 200 without `usage` block (edge / error shapes) → no emit - streaming 200 → no emit today (follow-up) - 4xx/5xx → no emit (no usage data to attribute) Tests (3 new): - `emits_usage_event_on_200_non_streaming_issue_404` — pins all 4 token counters, status_code, model_id, api_key_id, protocol - `skips_usage_event_when_upstream_omits_usage_block` — pins the no-emit edge for 200 without usage - `streaming_path_does_not_emit_today_but_passes_through` — pins the streaming MVP contract (no regression, no emission until the follow-up lands) References: - Parent: #226 (non-chat handlers don't emit) - Sibling MVP: #402 (embeddings) - Spec: <https://platform.openai.com/docs/api-reference/responses>
Warning Review limit reached
More reviews will be available in 9 minutes and 54 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe Responses API handler now emits usage telemetry from upstream successful responses. Internal structs ( ChangesUsage Telemetry Integration
Sequence DiagramsequenceDiagram
participant Client
participant ResponsesHandler
participant Dispatch
participant UpstreamAPI
participant UsageSink
participant OTLPExporter
Client->>ResponsesHandler: HTTP request
ResponsesHandler->>Dispatch: invoke dispatch
Dispatch->>UpstreamAPI: forward request
UpstreamAPI-->>Dispatch: response + usage block (non-streaming)
Dispatch->>Dispatch: extract usage counters
Dispatch-->>ResponsesHandler: ResponseDispatchSuccess{response, usage, model_id}
ResponsesHandler->>ResponsesHandler: emit AccessLog
ResponsesHandler->>ResponsesHandler: construct UsageEvent
ResponsesHandler->>UsageSink: publish UsageEvent
UsageSink->>OTLPExporter: fanout usage telemetry
ResponsesHandler-->>Client: HTTP response
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Note 🎁 Summarized by CodeRabbit FreeYour organization has reached its limit of developer seats under the Pro Plan. For new users, CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please add seats to your subscription by visiting https://app.coderabbit.ai/login.If you believe this is a mistake and have available seats, please assign one to the pull request author through the subscription management page using the link above. Comment |
5 tasks
…OW-1, LOW-2) PR #425 audit raised 2 MEDIUM + 2 LOW findings. All addressed: MEDIUM-1 — `extract_response_usage` returned `Some(zeros)` for `usage: {}` (malformed but present). Per the OpenAI Responses-API spec, `input_tokens` is required on every non-streaming 200, so its absence is upstream-malformed rather than a legitimate zero-spend reply. Tightened: gate emit on `input_tokens` being present and parseable; missing → no emit. New test `skips_usage_event_when_usage_block_is_empty_audit_m1` pins this. MEDIUM-2 — `upstream_error_returns_502` asserted only the status code, not the absence of a UsageEvent on the sink. A future regression that moved `emit_usage_event` into the error branch would silently ship. New test `upstream_5xx_does_not_emit_usage_event` adds the negative pinning. LOW-1 — Doc comment on `emit_usage_event` claimed completion / cached / reasoning were "left at Default::default()" but those are precisely the fields *populated*. Copy-paste bug from embeddings.rs (where they really were defaulted). Corrected to list cache_creation / cache_read / provider_request_id / etc. LOW-2 — Streaming pass-through test drained the body but didn't assert its shape. A regression that broke the byte-stream wiring in the new ResponseDispatchSuccess path could surface as 200 + empty body and still pass. Added `body_bytes.starts_with("data:")` to pin the SSE shape survives the refactor.
6 tasks
Uh oh!
There was an error while loading. Please reload this page.
moonming added a commit
that referenced
this pull request
May 27, 2026
PR #426 audit raised 3 MEDIUM + 3 LOW findings. Addressed inline; LOW-2 filed as #429 (cross-handler tightening that touches #425 too). MEDIUM-1 — Doc comment on `CompletionDispatchSuccess.model_id` referenced a non-existent `upstream_called` field (stale copy from embeddings.rs where it does exist). Corrected to describe the actual gating channel (`usage.is_some()`). MEDIUM-2 — 501 NotImplemented path was recording `status=200` in the access log and prometheus metrics (hardcoded `200u16`), making it impossible for operators to distinguish real successes from "provider does not support completions". Same systemic bug PR #404 and PR #405 fix in their own handlers. Switched to `success.response.status().as_u16()` + `RequestOutcome::from_status` to mirror the convention. UsageEvent emission was already correctly skipped via `usage: None`, so billing wasn't affected — only observability. MEDIUM-3 — No test exercised the 501 path. Added `provider_lacking_complete_returns_501_without_emit`: registers an `AnthropicBridge` (which doesn't override `Bridge::complete()` so the trait default returns `BridgeError::Config` → 501), routes a `/v1/completions` request at an Anthropic model, and pins both the 501 response status AND the absence of any UsageEvent on the sink. LOW-1 — Added `skips_usage_event_when_upstream_omits_usage_block_entirely` test that exercises the outer `body.get("usage")?` short-circuit in `extract_completion_usage` (the existing `*_when_upstream_usage_block_is_empty` test only covered the inner `prompt_tokens` missing case). LOW-3 — Comment referenced `#404` PR as if merged; on this branch it's now true (#425 landed) so the reference is correct. LOW-2 (gate `completion_tokens` on presence) deliberately deferred to #429 for cross-handler symmetry with #425's responses.rs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes#404 (non-streaming MVP). Sibling of #402 (embeddings).
Pre-fix, `/v1/responses` dropped the `UsageEvent` entirely. Every o1/o3/GPT-5 request via OpenAI's Responses API (the modern default entry for reasoning models) was invisible to cp-api's budget ledger and customer-facing /logs analytics — meaning customer spend on the most expensive model class wasn't being counted at all.
This PR emits a UsageEvent on every successful non-streaming 200 with:
Scope: non-streaming only
Streaming path is deferred. Responses-API streaming surfaces usage in the final `response.completed` SSE event, but the current handler passes the byte stream through verbatim and the gateway doesn't parse SSE chunks here. SSE byte-stream interception is its own design surface; tracked as a #404 streaming follow-up so the MVP can land and start capturing the majority of traffic (non-streaming).
Same MVP pattern PR #402 used for embeddings.
Emit semantics
Test plan
References
Summary by CodeRabbit
Release Notes