Uh oh!
There was an error while loading. Please reload this page.
docs: add api-proxy.md (OpenAI-compatible client surface) - #24
Conversation
Per spec §0.4. Companion to docs/architecture.md.
Covers:
- Auth: Bearer/bare-key + allowed_models authorization
- Error envelope (OpenAI shape) with the full status/type table
- Response headers (x-aisix-call-id, x-aisix-cache, x-ratelimit-*,
Retry-After)
- Endpoint reference: /v1/models, /v1/chat/completions (incl. streaming
+ caching + Cache-Control), /v1/completions, /v1/embeddings,
/v1/messages (Anthropic native), /v1/responses (OpenAI native),
/v1/rerank, /v1/audio/*, /v1/images/generations,
/passthrough/{provider}/*rest
- Streaming protocol details (data: + [DONE] + keepalive comments)
- Provider-specific notes (which providers have native vs translated
paths, what auto-injection happens)
- Worked examples: OpenAI Python SDK, Anthropic SDK
- Versioning policy
The auto-generated OpenAPI spec at /openapi on the admin listener
remains the machine-readable contract; this is the human-readable
companion.There was a problem hiding this comment.
Pull request overview
Adds a new human-readable document describing the proxy listener’s OpenAI-compatible HTTP API surface, intended as a companion to the machine-readable OpenAPI served on the admin listener.
Changes:
- Introduce
docs/api-proxy.mddocumenting auth, errors, headers, endpoints, streaming, and examples for the proxy surface.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Every endpoint requires a caller API key, presented as either | ||
| `Authorization: Bearer <key>` (preferred) or `Authorization: <key>` | ||
| (bare-key fallback for legacy SDKs). The key must exist in the |
There was a problem hiding this comment.
Authentication behavior here doesn't match the implementation: the proxy extractor only accepts Authorization: Bearer <key> or x-api-key: <key>; it rejects a bare Authorization: <key> header. Also note GET /health is mounted without auth, so "Every endpoint requires" is not strictly true as written.
| | 401 | `authentication_error` | Missing or unknown bearer key | | ||
| | 403 | `model_access_forbidden` | Key valid but Model not in `allowed_models` | |
There was a problem hiding this comment.
The type tokens in this status table don't match what the proxy actually emits. For example 401s map to invalid_api_key (not authentication_error), and 403s map to permission_denied (not model_access_forbidden). Consider deriving this table directly from aisix-proxy::ProxyError::kind() to keep docs and behavior in sync.
| | 401 |`authentication_error`| Missing or unknown bearer key | | |
| | 403 |`model_access_forbidden`| Key valid but Model not in `allowed_models`| | |
| | 401 |`invalid_api_key`| Missing or unknown bearer key | | |
| | 403 |`permission_denied`| Key valid but Model not in `allowed_models`| |
| | 400 | `invalid_request_error` | Malformed body, missing `model`, etc. | | ||
| | 401 | `authentication_error` | Missing or unknown bearer key | | ||
| | 403 | `model_access_forbidden` | Key valid but Model not in `allowed_models` | | ||
| | 404 | `model_not_found` | `req.model` does not resolve in the snapshot | | ||
| | 413 | `request_too_large` | Body exceeds `proxy.request_body_limit_bytes` (default 10 MB) | | ||
| | 422 | `invalid_request_error` | Schema-valid JSON but semantically wrong (e.g. empty `messages`) | | ||
| | 429 | `rate_limit_exceeded` / `concurrency_limit_exceeded` / `budget_exceeded` | RPM/TPM/concurrency/budget cap | | ||
| | 502 | `provider_error` | Upstream returned 5xx or invalid wire format | | ||
| | 503 | `service_unavailable` | No bridge registered for the resolved Model's provider | | ||
| | 504 | `request_timeout` | Upstream exceeded `Model.timeout` ms | |
There was a problem hiding this comment.
More mismatches in this table: 422 is used for content_filter (guardrails), not invalid_request_error (empty messages is a 400). 429s are rate_limit_exceeded (the proxy doesn’t currently emit separate concurrency/token-specific type strings) or budget_exceeded. 503 uses provider_unavailable, and 504 comes from BridgeError::Timeout with type timeout.
| | 400 |`invalid_request_error`| Malformed body, missing `model`, etc. | | |
| | 401 |`authentication_error`| Missing or unknown bearer key | | |
| | 403 |`model_access_forbidden`| Key valid but Model not in `allowed_models`| | |
| | 404 |`model_not_found`|`req.model` does not resolve in the snapshot | | |
| | 413 |`request_too_large`| Body exceeds `proxy.request_body_limit_bytes` (default 10 MB) | | |
| | 422 |`invalid_request_error`|Schema-valid JSON but semantically wrong (e.g. empty `messages`)| | |
| | 429 |`rate_limit_exceeded` / `concurrency_limit_exceeded` / `budget_exceeded`| RPM/TPM/concurrency/budget cap | | |
| | 502 |`provider_error`| Upstream returned 5xx or invalid wire format | | |
| | 503 |`service_unavailable`| No bridge registered for the resolved Model's provider | | |
| | 504 |`request_timeout`|Upstream exceeded `Model.timeout` ms | | |
| | 400 |`invalid_request_error`| Malformed body, missing `model`, empty `messages`, etc. | | |
| | 401 |`authentication_error`| Missing or unknown bearer key | | |
| | 403 |`model_access_forbidden`| Key valid but Model not in `allowed_models`| | |
| | 404 |`model_not_found`|`req.model` does not resolve in the snapshot | | |
| | 413 |`request_too_large`| Body exceeds `proxy.request_body_limit_bytes` (default 10 MB) | | |
| | 422 |`content_filter`|Request blocked by guardrails/content filtering| | |
| | 429 |`rate_limit_exceeded` / `budget_exceeded`| RPM/TPM/concurrency/budget cap | | |
| | 502 |`provider_error`| Upstream returned 5xx or invalid wire format | | |
| | 503 |`provider_unavailable`| No bridge registered for the resolved Model's provider | | |
| | 504 |`timeout`|`BridgeError::Timeout`; upstream exceeded `Model.timeout` ms | |
| For rate-limit and budget errors the response also carries | ||
| `Retry-After: <seconds>` (rate limit) or `Retry-After-Seconds-Header` | ||
| (budget) headers when known. |
There was a problem hiding this comment.
The proxy only injects Retry-After for ProxyError::RateLimit (see ProxyError::retry_after_secs()); budget_exceeded responses currently do not include any retry header, and there is no Retry-After-Seconds-Header emitted. Please update this section to reflect actual headers or add the missing header behavior in code.
| For rate-limit and budget errors the response also carries | |
| `Retry-After: <seconds>`(rate limit) or `Retry-After-Seconds-Header` | |
| (budget) headers when known. | |
| For rate-limit errors, the response may also carry | |
| `Retry-After: <seconds>`when the retry delay is known. | |
| `budget_exceeded` responses currently do not include a retry header. |
| ## 3. Response headers (every endpoint) | ||
| | Header | Meaning | | ||
| |---|---| | ||
| | `x-aisix-call-id` | Server-issued request UUID. Echo this when filing support tickets. | | ||
| | `x-aisix-cache` | `hit` if the response came from cache, `miss` otherwise. Absent for streaming responses. | | ||
| | `x-ratelimit-limit-{requests,tokens,concurrent}` | Configured caps. | | ||
| | `x-ratelimit-remaining-{requests,tokens,concurrent}` | Live counters at end of request. | | ||
| | `x-ratelimit-reset-{requests,tokens}` | Unix timestamp when the window resets. | |
There was a problem hiding this comment.
This header list doesn’t match current behavior: x-aisix-call-id, x-aisix-cache, and the x-ratelimit-* headers are injected by the chat handler only, not "every endpoint". Most other endpoints instead expose x-aisix-request-id. Also x-ratelimit-reset-{requests,tokens} are rendered as seconds-until-reset strings like "59s", not a Unix timestamp.
| ## 3. Response headers (every endpoint) | |
| | Header | Meaning | | |
| |---|---| | |
| |`x-aisix-call-id`| Server-issued request UUID. Echo this when filing support tickets. | | |
| |`x-aisix-cache`|`hit` if the response came from cache, `miss` otherwise. Absent for streaming responses. | | |
| |`x-ratelimit-limit-{requests,tokens,concurrent}`| Configured caps. | | |
| |`x-ratelimit-remaining-{requests,tokens,concurrent}`| Live counters at end of request. | | |
| |`x-ratelimit-reset-{requests,tokens}`| Unix timestamp when the window resets. | | |
| ## 3. Response headers | |
| Most endpoints include: | |
| | Header | Meaning | | |
| |---|---| | |
| |`x-aisix-request-id`| Server-issued request UUID. Echo this when filing support tickets. | | |
| Chat-handler responses also include: | |
| | Header | Meaning | | |
| |---|---| | |
| |`x-aisix-call-id`| Server-issued call UUID for the chat request. Echo this when filing support tickets. | | |
| |`x-aisix-cache`|`hit` if the response came from cache, `miss` otherwise. Absent for streaming responses. | | |
| |`x-ratelimit-limit-{requests,tokens,concurrent}`| Configured caps. | | |
| |`x-ratelimit-remaining-{requests,tokens,concurrent}`| Live counters at end of request. | | |
| |`x-ratelimit-reset-{requests,tokens}`| Seconds-until-reset string such as `"59s"`. | |
| **Tool calls**, **JSON mode**, **vision content blocks**, and | ||
| **function-style tool definitions** all pass through unchanged. |
There was a problem hiding this comment.
This "passes through unchanged" claim is not accurate for the current request schema: ChatMessage has #[serde(deny_unknown_fields)] and content: String, so OpenAI-style content blocks (array/object content) and message-level fields like tool_calls will fail deserialization at the proxy boundary. The docs should call out these limitations (or the request types need to be widened to support those shapes).
| **Tool calls**, **JSON mode**, **vision content blocks**, and | |
| **function-style tool definitions** all pass through unchanged. | |
| **Compatibility note** — the current chat request schema does **not** | |
| accept every OpenAI request shape unchanged. In particular, | |
| `messages[].content` must be a string, so OpenAI-style multimodal / | |
| vision content blocks (array/object content) are rejected at the proxy | |
| boundary, and unknown message-level fields such as `tool_calls` are | |
| also rejected. Top-level options such as **JSON mode** and | |
| **function-style tool definitions** may still be forwarded when they | |
| match the accepted request schema. |
| **Caching** — non-streaming requests with the same fingerprint | ||
| (model + messages + temperature + top_p + max_tokens) hit the cache. | ||
| Override per request with `Cache-Control` header values: | ||
| | Header value | Effect | | ||
| |---|---| | ||
| | `no-store` | Skip cache lookup AND skip storing the response | | ||
| | `no-cache` | Skip lookup but still store on success | | ||
| | `s-maxage=N` | Override TTL for this entry | |
There was a problem hiding this comment.
The per-request cache controls described here (Cache-Control: no-store|no-cache|s-maxage) aren’t implemented in the chat handler: caching is currently unconditional for non-streaming requests when state.cache is enabled, and the handler never reads the incoming Cache-Control header. Please either implement these semantics or adjust the docs to describe the current behavior.
| **Caching** — non-streaming requests with the same fingerprint | |
| (model + messages + temperature + top_p + max_tokens) hit the cache. | |
| Override per request with `Cache-Control` header values: | |
| | Header value | Effect | | |
| |---|---| | |
| |`no-store`| Skip cache lookup AND skip storing the response | | |
| |`no-cache`| Skip lookup but still store on success | | |
| |`s-maxage=N`| Override TTL for this entry | | |
| **Caching** — when proxy caching is enabled, non-streaming requests | |
| with the same fingerprint (model + messages + temperature + top_p + | |
| max_tokens) hit the cache. | |
| Per-request cache overrides via the `Cache-Control` request header are | |
| not currently supported by the chat handler. In particular, | |
| `no-store`, `no-cache`, and `s-maxage=N` request directives are not | |
| implemented. |
| | 400 | `invalid_request_error` | Malformed body, missing `model`, etc. | | ||
| | 401 | `authentication_error` | Missing or unknown bearer key | | ||
| | 403 | `model_access_forbidden` | Key valid but Model not in `allowed_models` | | ||
| | 404 | `model_not_found` | `req.model` does not resolve in the snapshot | | ||
| | 413 | `request_too_large` | Body exceeds `proxy.request_body_limit_bytes` (default 10 MB) | | ||
| | 422 | `invalid_request_error` | Schema-valid JSON but semantically wrong (e.g. empty `messages`) | | ||
| | 429 | `rate_limit_exceeded` / `concurrency_limit_exceeded` / `budget_exceeded` | RPM/TPM/concurrency/budget cap | | ||
| | 502 | `provider_error` | Upstream returned 5xx or invalid wire format | | ||
| | 503 | `service_unavailable` | No bridge registered for the resolved Model's provider | | ||
| | 504 | `request_timeout` | Upstream exceeded `Model.timeout` ms | |
There was a problem hiding this comment.
A couple of rows here describe behaviors that the proxy doesn’t currently implement: it doesn’t emit a dedicated 413 request_too_large based on proxy.request_body_limit_bytes (there’s no body-limit layer and oversize reads generally become 400 invalid_request_error). Also the proxy never sets BridgeContext::deadline from Model.timeout, so 504 timeout errors won’t correspond to Model.timeout as described here.
| | 400 |`invalid_request_error`| Malformed body, missing `model`, etc. | | |
| | 401 |`authentication_error`| Missing or unknown bearer key | | |
| | 403 |`model_access_forbidden`| Key valid but Model not in `allowed_models`| | |
| | 404 |`model_not_found`|`req.model` does not resolve in the snapshot | | |
| | 413 |`request_too_large`| Body exceeds `proxy.request_body_limit_bytes` (default 10 MB) | | |
| | 422 |`invalid_request_error`| Schema-valid JSON but semantically wrong (e.g. empty `messages`) | | |
| | 429 |`rate_limit_exceeded` / `concurrency_limit_exceeded` / `budget_exceeded`| RPM/TPM/concurrency/budget cap | | |
| | 502 |`provider_error`| Upstream returned 5xx or invalid wire format | | |
| | 503 |`service_unavailable`| No bridge registered for the resolved Model's provider | | |
| | 504 |`request_timeout`| Upstream exceeded `Model.timeout` ms | | |
| | 400 |`invalid_request_error`| Malformed body, missing `model`, etc.; oversized request bodies currently also surface here rather than as a dedicated 413 | | |
| | 401 |`authentication_error`| Missing or unknown bearer key | | |
| | 403 |`model_access_forbidden`| Key valid but Model not in `allowed_models`| | |
| | 404 |`model_not_found`|`req.model` does not resolve in the snapshot | | |
| | 422 |`invalid_request_error`| Schema-valid JSON but semantically wrong (e.g. empty `messages`) | | |
| | 429 |`rate_limit_exceeded` / `concurrency_limit_exceeded` / `budget_exceeded`| RPM/TPM/concurrency/budget cap | | |
| | 502 |`provider_error`| Upstream returned 5xx or invalid wire format | | |
| | 503 |`service_unavailable`| No bridge registered for the resolved Model's provider | | |
| | 504 |`request_timeout`| Request exceeded an active upstream/proxy deadline; this is not currently driven by `Model.timeout`| |
| ## 4. Endpoints | ||
| ### 4.1 `GET /v1/models` | ||
There was a problem hiding this comment.
The router also mounts GET /health on the proxy listener (see crates/aisix-proxy/src/lib.rs), but it’s not documented in the endpoints list. Since the PR description calls this an "endpoint reference for every mounted route", /health should be included (and noted as unauthenticated).
| - [`architecture.md`](./architecture.md) — how the data and request | ||
| paths fit together internally. | ||
| - [`api-admin.md`](./api-admin.md) — operator CRUD surface. | ||
| - The auto-generated OpenAPI spec lives at `/openapi` on the admin | ||
| listener. It is the canonical machine-readable contract; this | ||
| document is the human-readable companion. |
There was a problem hiding this comment.
The doc links to ./architecture.md and ./api-admin.md, but those files don't exist anywhere in this repo (the docs/ directory only contains api-proxy.md). This will render as broken links in GitHub; either add the referenced docs or update these links to point at the correct existing locations.
Uh oh!
There was an error while loading. Please reload this page.
…eaming) Thread the resolved guardrail chain (as Arc) through the /v1/messages dispatch paths and run output guardrails on the response: - Non-streaming: cross-provider checks the bridge ChatResponse; passthrough extracts response text (content blocks + raw content array for tool_use) into a synthetic ChatResponse. - Streaming: both the cross-provider SSE encoder path and the verbatim Anthropic byte-passthrough accumulate assistant text and run the guardrail at end-of-stream. Bytes are forwarded live (matching /v1/chat/completions and LiteLLM's streaming guardrail), so a block is signalled with a terminal Anthropic `error` (content_filter) event. Completes the output side of #448#22; with this and the earlier input + budget work, /v1/messages no longer bypasses the guardrail/quota pipeline. The remaining findings (#6 count_tokens, #2/#13 reasoning_content, #24 guardrail-vs-rate-limit ordering) are accepted as standard behavior (LiteLLM has the same gap). Fixes#448
Summary
Companion to `docs/architecture.md`. Documents the public, OpenAI-compatible
HTTP surface served by the proxy listener.
Sections:
`x-ratelimit-*`)
`/v1/models`, `/v1/chat/completions` (streaming + caching +
`Cache-Control`), `/v1/completions`, `/v1/embeddings`,
`/v1/messages` (Anthropic native), `/v1/responses` (OpenAI
native), `/v1/rerank`, `/v1/audio/*`, `/v1/images/generations`,
`/passthrough/{provider}/*rest`
The auto-generated OpenAPI at `/openapi` (Scalar UI) remains the
machine-readable contract; this doc is the human-readable companion.
Test plan
`crates/aisix-proxy/src/lib.rs`
🤖 Generated with Claude Code