Uh oh!
There was an error while loading. Please reload this page.
feat: ensemble model — panel + judge fan-out dispatch (config + non-streaming) - #604
Merged
Conversation
Introduce a third virtual-model kind alongside direct and routing: an `ensemble` model fans a chat request out to a panel of models and synthesizes their responses via a judge model. This lands the config layer only; DP fan-out/synthesis dispatch is a follow-up. - New EnsembleConfig/PanelMember/Judge types (models/ensemble.rs), mirroring the routing config shape (deny_unknown_fields, _or_default accessors, per-member temperature/seed for self-ensemble diversity). - Model.ensemble field + Model::is_ensemble(). - Runtime validator (models/schema.rs): add the ensemble block and widen the direct-vs-routing oneOf to a three-way XOR (direct | routing | ensemble), so an ensemble row is neither silently dropped on the kine watch path nor allowed to mix shapes. - Regenerate schemas; emit a standalone ensemble.schema.json for CP consumption (parity with routing.schema.json). Refs #601, api7/AISIX-Cloud#804.
Wire the ensemble virtual-model kind end-to-end (non-streaming). One /v1/chat/completions request to an `ensemble` model fans out to its panel of models in parallel, then a judge model synthesizes a single answer. - ensemble.rs: pure executor `run_ensemble` (parallel fan-out, min_responses gate, per-member temperature/seed for self-ensemble diversity, judge synthesis with neutral candidate labels + per-candidate truncation, judge retry-once-on-transient), plus the production `ProxyModelCaller` that resolves each member display_name -> ProviderKey -> Bridge. - chat.rs: `dispatch_ensemble` branch (after the entry-level rate-limit reservation, before the failover/streaming machinery): rejects tools + streaming with 400; runs the output guardrail on the synthesized answer; commits the aggregate panel+judge tokens once; emits one usage event per sub-call (attempt_kind panel/judge) sharing request_id, before the guardrail check so a blocked response still bills the full fan-out; suppresses the entry-level usage event to avoid double-emit. - Client sees the ensemble's own model name; panel/judge model names never reach the client (synthesized answer, response.model, or error messages). - Tests: 8 executor unit tests + 5 handler e2e (fan-out->judge, tools/stream 400, insufficient-panel->502, output-block-still-bills-panel+judge). Streaming is rejected for now (no single-response->SSE path); it lands in a follow-up. Dashboard needs to recognize the new attempt_kind values "panel"/"judge" (free-string on the wire; tracked CP-side). Refs #601, api7/AISIX-Cloud#804.
ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: 📒 Files selected for processing (7)
🚧 Files skipped from review as they are similar to previous changes (3)
📝 WalkthroughWalkthroughAdds "ensemble" model support across the proxy stack: new ChangesEnsemble Model Feature
Sequence Diagram(s)sequenceDiagram
participant Client
participant dispatch as chat.rs<br/>dispatch
participant dispatch_ensemble as dispatch_ensemble
participant run_ensemble as run_ensemble
participant ProxyModelCaller as ProxyModelCaller
rect rgba(70, 130, 180, 0.5)
Note over dispatch,ProxyModelCaller: Panel fan-out
Client->>dispatch: POST /v1/chat/completions<br/>(ensemble model)
dispatch->>dispatch: is_ensemble() check
dispatch->>dispatch_ensemble: validate tools,<br/>stream, ensemble config
dispatch_ensemble->>run_ensemble: run_ensemble(req, config, caller)
run_ensemble->>ProxyModelCaller: call(panel_member, panel_request) ×N
ProxyModelCaller-->>run_ensemble: collect successes
run_ensemble->>run_ensemble: enforce<br/>min_responses_or_default()
end
rect rgba(60, 179, 113, 0.5)
Note over run_ensemble,ProxyModelCaller: Judge synthesis with retry
run_ensemble->>run_ensemble: build judge_request<br/>with labeled candidates
run_ensemble->>ProxyModelCaller: call_judge_with_retry(judge)
ProxyModelCaller-->>run_ensemble: EnsembleOutcome
end
rect rgba(220, 100, 60, 0.5)
Note over dispatch_ensemble,Client: Guardrails + telemetry
dispatch_ensemble->>dispatch_ensemble: emit UsageEvents<br/>per panel member & judge
dispatch_ensemble->>dispatch_ensemble: evaluate output guardrails
dispatch_ensemble-->>Client: synthesized ChatResponse<br/>(or 400/502/ContentFiltered)
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Note 🎁 Summarized by CodeRabbit FreeYour organization has reached its limit of developer seats under the Pro Plan. For new users, CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please add seats to your subscription by visiting https://app.coderabbit.ai/login.If you believe this is a mistake and have available seats, please assign one to the pull request author through the subscription management page using the link above. Comment |
An independent audit of #604 found a client-facing leak, a CI-blocking schema drift, and billing gaps on the ensemble error exits. - Redact ensemble member identity from client errors: a misconfigured judge's display_name + provider_key_id no longer reach error.message (the resolve_provider_key path still forwarded it). Detail kept in logs. - Regenerate ensemble/model schema JSON so the schema-drift CI gate passes (the timeout_ms doc edit hadn't been regenerated). - Bill the panel on every error exit, not just success. Panel members that already round-tripped upstream now emit usage + commit tokens on the insufficient-panel (502) and judge-failure paths, matching the output-guardrail-block path. EnsembleError carries the surviving PanelOutcomes. - tools: [] (empty) no longer 400s; only a non-empty tools array or a forcing tool_choice rejects (many SDKs always send tools: []). - Non-chat endpoints reject an ensemble model with an explicit, accurate 400 (was incidental and said "routing"); added coverage. - Remove dead EnsembleConfig::is_empty(). Tests: +6 e2e (judge-error redaction, 502-path + judge-failure panel billing, tools:[] accepted, non-chat 400). 259 core + 439 proxy green; clippy -D warnings clean; schema-drift gate clean. Refs #601.
moonming
commented
Jun 15, 2026
MemberAuthor
Post-PR independent audit — all findings resolved ( |
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First slice of the ensemble virtual-model feature (DP side, tracked in #601; CP umbrella api7/AISIX-Cloud#804). An
ensemblemodel fans one/v1/chat/completionsrequest out to a panel of N upstream models in parallel, then a judge model synthesizes a single answer. It extends the existing virtual-model machinery:routingpicks one target;ensemblecalls all and synthesizes.This PR lands the config layer and the non-streaming dispatch. Streaming, per-target rate-limiting, and hardening are separate follow-ups (see Deferred) — so this does notclose#601.
What's in this PR
1a89a6e— config layer (newensemblemodel kind)EnsembleConfig { panel: [{model, temperature?, seed?, weight?}], judge: {model, synthesis_prompt?}, min_responses?, timeout_ms? }inaisix-core.Model.ensemble+Model::is_ensemble()(DP convention: virtual-model kind = presence of the block, same asrouting).schemas/resources/model.schema.json(+ a standaloneensemble.schema.json) and the hand-written runtime validatormodels/schema.rs::model_schema(), whoseoneOfis now a 3-way XOR (direct | routing | ensemble).8375c85— non-streaming fan-out dispatchensemble.rs: pure executorrun_ensemble(paralleljoin_allfan-out,min_responsesgate, per-membertemperature/seedoverride for self-ensemble diversity, judge synthesis with neutralAnswer 1..Nlabels + per-candidate truncation, judge retry-once-on-transient), plus the productionProxyModelCallerthat resolves each member → ProviderKey → Bridge via the existing routing helpers.chat.rs::dispatch_ensemble: branches at the post-rate-limit-reservation seam; rejectstools/tool_choiceandstream:truewith 400; runs the output guardrail on the synthesized answer; commits the aggregate panel+judge tokens once against the single entry-level reservation; emits one usage event per sub-call (attempt_kind=panel/judge, sharingrequest_id) before the guardrail check so a blocked response still bills the full fan-out; suppresses the entry-level usage event (no double-emit).response.model, or error messages).Behavior / contract
tools/streaming rejected (see Deferred).Deferred (separate PRs, tracked in #601)
stream:trueis 400-rejected here; there is no single-ChatResponse→SSE path today, so it's net-new.attempt_kindvaluespanel/judge(free-string on the wire — won't break ingestion).Independent audit (merge-gate)
An independent cold-review agent audited this change. It confirmed: no double-emit / double-commit; authorization consistent with routing (panel/judge are internal targets — no escalation beyond what routing already allows); sound snapshot lifetimes across the fan-out; model names not leaked to the client response body. It caught and we fixed:
charge: Noneon block) and locked withensemble_output_block_still_emits_panel_and_judge_usage.error.message→ redacted (detail kept in server logs).timeout_msdoc/behavior mismatch → behavior made uniform (now also applies to the judge call) + doc corrected.Test plan
cargo test -p aisix-core— 259 pass (config types, runtime-validator 3-way XOR, schema round-trip; e.g.model_ensemble_with_direct_fields_fails,model_ensemble_with_routing_fails).cargo test -p aisix-proxy— 433 pass: 8 executor unit tests + 5 handler e2e through a realOpenAiBridgeagainst wiremock:ensemble_fans_out_to_panel_and_returns_judge_synthesis(judge answer returned,model == "council", no upstream-id leak)ensemble_rejects_tool_requests_with_400,ensemble_rejects_streaming_with_400ensemble_insufficient_panel_returns_502ensemble_output_block_still_emits_panel_and_judge_usage(HIGH regression lock — asserts 2 panel + 1 judge usage events still fire on a block)cargo clippy --all-targets -- -D warnings— clean; full workspace builds.Summary by CodeRabbit
New Features
Bug Fixes