Uh oh!
There was an error while loading. Please reload this page.
feat(provider-vertex): add aisix-provider-vertex skeleton crate (D5, #302 §3 Phase E) - #312
Conversation
…Phase E / D5) Wave 5 D5 — scaffolds the Google Vertex AI family bridge so a `provider_key` row with `adapter: "vertex"` resolves to a real bridge in the Hub instead of falling through to the legacy fallback. The actual GCP OAuth + per-publisher HTTP dispatch is tracked as D5.1-D5.5 follow-ups; this PR ships the: - new `aisix-provider-vertex` crate registered in the workspace - `VertexBridge` struct implementing `aisix_gateway::Bridge` with a clear `BridgeError::Config` returning the tracking issue # on every chat / chat_stream call (skeleton contract — an operator who reaches this path knows it's intentional WIP, not silently broken) - `VertexPublisher` enum (Google / Anthropic / Meta / Mistral / Ai21) with `from_upstream_id()` and `url_segment()` helpers per the LiteLLM `vertex_ai/` multi-publisher single-entry pattern — a single `google-vertex` catalog provider dispatches by model prefix at runtime inside the bridge instead of forcing customers to register one provider_key per publisher even though the GCP credential is shared - Hub registration via `hub.register_family(Adapter::Vertex, ...)` in `build_hub()` — the new two-tier dispatch wire (D1) can now see Vertex; the legacy `Provider`-keyed path is unchanged Tests (10 passing): - publisher prefix resolution for gemini / claude / llama / mistral / codestral / jamba - case-insensitive matching on the model name - unknown prefix returns None (drives the publisher-resolution guard in chat / chat_stream) - url_segment matches Vertex API path conventions (Mistral's Vertex publisher tag is `mistralai`, not `mistral` — pinned to catch a typo regression) - chat() surfaces clear "not yet implemented + issue #302" error - chat() with unknown publisher prefix errors before dispatch - bridge.name() == "vertex" (metrics label stability) - wire.rs reserved query params (`alt`, `key`, `access_token`) covered for the eventual default-headers / default-body-fields override guard References (per CLAUDE.md §7): - LiteLLM vertex_ai/ design — single prefix, internal publisher dispatch: https://github.com/BerriAI/litellm/tree/main/litellm/llms/vertex_ai - Vertex AI REST API — https://cloud.google.com/vertex-ai/docs/reference/rest - Vertex publishers index — https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models Out of scope, tracked under #302 Phase E follow-ups: - D5.1 google-cloud-auth / yup-oauth2 bearer token acquisition (service account JSON key, ADC, metadata server) - D5.2 Gemini publisher dispatch (`:streamGenerateContent`) - D5.3 Anthropic-on-Vertex dispatch (`:streamRawPredict`) - D5.4 Llama / Mistral / AI21 publisher dispatch - D5.5 BridgeContext.deadline + Retry-After plumbing
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (7)
Note 🎁 Summarized by CodeRabbit FreeYour organization has reached its limit of developer seats under the Pro Plan. For new users, CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please add seats to your subscription by visiting https://app.coderabbit.ai/login.If you believe this is a mistake and have available seats, please assign one to the pull request author through the subscription management page using the link above. Comment |
… LOW-2
HIGH-1 — Meta.url_segment() pointed at non-existent path. LiteLLM's
vertex_llm_base.py:277 routes Llama through the OpenAPI shim at
`endpoints/openapi/chat/completions`, NOT `publishers/meta/...`.
Returning the latter would 404 on every Llama-on-Vertex request once
dispatch lands.
Fix: change url_segment() return to Option<&'static str> and return
None for Meta. The D5.4 follow-up will build the OpenAPI-shim URL
separately via its own code path rather than synthesizing the wrong
publishers/meta/ form.
HIGH-2 — `llama-` prefix didn't match real Vertex Llama model ids.
LiteLLM vertex_ai_partner_models/main.py:33-34 recognizes Meta via
`META_PREFIX = "meta/"` and the bare-`llama` family. Real Vertex
model ids are `meta/llama-3.3-70b-instruct-maas` and
`llama3-405b-instruct-maas` (note: `llama3-` has a digit immediately
after `llama`, not a hyphen). The `lower.starts_with("llama-")`
matcher rejected both.
Fix: match `lower.starts_with("meta/") || lower.starts_with("llama")`.
Tests updated with the real wire forms.
MEDIUM-1 — Doc comment claimed "follows the LiteLLM vertex_ai/
convention" without acknowledging the gap. LiteLLM also routes
deepseek-ai/*, qwen*, openai/gpt-oss-*, minimaxai/*, moonshotai/*,
zai-org/*. We don't handle any of these today; the operator gets a
clear "publisher unknown" error if they try.
Fix: amended the doc comment to enumerate which prefixes ARE handled
and which are known-missing-for-follow-up. The "publisher unknown"
contract is still correct — just documented honestly now.
MEDIUM-2 — Cutover risk between cp-api enabling `google-vertex` on
the catalog and D5.2 (Gemini dispatch) shipping. Today cp-api's
`isSupportedProvider` gate blocks `provider: "google-vertex"` at
createProviderKey, so the skeleton bridge is unreachable in
production. The moment Phase B loosens that gate, every chat through
a google-vertex PK lands on this NOT-IMPLEMENTED skeleton.
Fix: explicit CUTOVER CAUTION comment in `build_hub()` documenting
the strict order — D5.2 must merge before the cp-api catalog flip.
LOW-2 — `let _ = wire::reserved_query_params()` code smell. Kept
as-is: tests exercise the function directly, but the explicit
reachability in chat() makes the future dispatch PR's diff cleaner
(no "function added but unused" warning when the override apply
path is wired in). Justified inline.moonming
commented
May 17, 2026
Audit response — 2 HIGH + 2 MEDIUM + 1 LOWFixed in `50cd6d9`
Kept as justified (LOW-1, LOW-2)
Test results``` All HIGH and MEDIUM either fixed or explicitly justified per CLAUDE.md §8 merge gate. |
…el.model_name) D6 audit on PR #313 surfaced a wire-shape bug that applies equally to D5's skeleton: chat() and chat_stream() resolved the publisher from req.model (customer-typed display name) instead of from ctx.model.model_name (operator-pinned upstream id). Once dispatch lands in D5.2 (Gemini publisher), the URL builder would have produced `.../publishers/google/models/<customer-facing- name>:streamGenerateContent` and 404 on every Vertex request. The skeleton tests previously pinned the wrong contract (passed `req.model="gemini-1.5-pro"` so the matcher accidentally got the right input), so a future dispatch PR would have inherited the broken wire. Fix: - Add upstream_model(ctx) helper mirroring OpenAiBridge. - chat() / chat_stream() resolve from ctx.model.model_name. - req.model is now _req and explicitly ignored. - sample_model() renamed to sample_model_with(model_name) so tests can pin "display_name differs from model_name" by construction. - New regression test chat_ignores_req_model_and_uses_ctx_model_name sets req.model="gpt-4o" (would fail publisher resolution if it were the source of truth) and asserts the not-implemented stub fires — proving model_name was the actual input. - New defense test chat_with_missing_model_name_errors_before_dispatch ensures Option<String> = None on Model.model_name surfaces a clear error rather than panicking. D6 (#313) already fixed; D7 (#314) was authored after D6's audit landed so it already has the fix.
moonming
commented
May 17, 2026
Backport: D6 audit HIGH-1 (`req.model` → `ctx.model.model_name`)D6's audit on PR #313 surfaced a wire-shape bug that applies equally to D5's skeleton — the initial D5 audit missed it. `chat()` / `chat_stream()` resolved the publisher from `req.model` (customer-typed display name in `/v1/chat/completions`) instead of `ctx.model.model_name` (operator-pinned upstream id). Once D5.2 lands, the URL builder would have produced `.../publishers/google/models/:streamGenerateContent` and 404 on every Vertex request. Fixed in `acf779e`
Test results``` D6 (#313) already fixed; D7 (#314) was authored after D6's audit so it already had the fix. |
Uh oh!
There was an error while loading. Please reload this page.
Summary
Phase E / D5 of api7/AISIX-Cloud#302 — scaffolds the Google Vertex AI family bridge so a `provider_key` row with `adapter: "vertex"` resolves to a real bridge in the Hub.
Skeleton PR: the crate exists, compiles, is registered in the workspace + Hub. The actual GCP OAuth + per-publisher HTTP dispatch is tracked as D5.1–D5.5 follow-ups (see crate-level docs in `lib.rs`).
What lands
Why multi-publisher single entry (not per-publisher provider)
Diverging from LiteLLM would force every customer to register a separate provider_key per publisher even though the GCP credential is the same — exactly the operator pain `google-vertex` solves. The publisher is encoded in the upstream model id (e.g. `claude-3-5-sonnet@20241022`), so the bridge can dispatch internally without exposing the split to cp-api / dashboard.
Tests (10 passing)
References (per CLAUDE.md §7)
Out of scope — D5 follow-up tracker
Test plan
Related