feat: model knobs, LiteLLM-verbatim chat lane, per-door passthrough params by rejojer · Pull Request #409 · VectifyAI/PageIndex

rejojer · 2026-08-16T16:42:06Z

Ports the pending feat/local-chat work to main (16 commits; per-commit history stays on feat/local-chat, review record in #400).

Model knobs. The documented surface becomes index_model (indexing: structure + summaries) + chat_model (all chat doors), resolved in one seam (ConfigLoader.load()), with every released name still accepted: model sets both roles, summary_model/retrieve_model keep their specific roles. New names win over old, specific over general, and the packaged config.yaml ships no model keys, so key presence distinguishes user choice from built-in defaults (gpt-5.6-luna / gpt-5.6-sol). CLI leads with --index-model (--model stays as legacy synonym); the standard branch's silently-ignored --summary-model now works.

Chat lane routing. Every chat-lane model routes through LiteLLM — bare names are OpenAI-compatible shorthand, and no prefix triggers a direct lane; the model-name grammar is LiteLLM's, verbatim. responses() stays native.

Per-door passthrough. Protocol-native reasoning params, forwarded verbatim with no default of ours: chat_completions(reasoning_effort=), responses(reasoning={}), messages(thinking={}). Named top_p/max_tokens on the chat door and max_output_tokens on responses (ModelSettings fields — the one channel clean on every lane; both caps bound each backend call in the loop, like max_turns bounds the loop). An extra_body escape hatch on all three doors for fields the methods don't name, merged last so caller keys win. Cloud rejects the local-only knobs explicitly.

litellm floor 1.84 → 1.97.0 — the release whose bridge routes sol-class chatcmpl+tools calls onto /v1/responses automatically (A/B-verified against 1.96.2), so the default chat model works out of the box.

285 tests green; wire-level probes verified max_tokens translation on both LiteLLM paths (chatcmpl max_completion_tokens, bridge max_output_tokens), extra_body delivery per lane, and the anthropic runner spreading extra_body into every turn's request.

… now

…ncluded The direct-OpenAI special case existed to dodge LiteLLM's import cost, and it made OpenAI's own Responses-first models fail on the front door: gpt-5.6-sol 400s on chatcmpl+tools while reasoning is on (server-side policy — wire-captured with no reasoning_effort in our request). LiteLLM 1.97 translates such calls onto /v1/responses; 1.84 does not, so the sol-class 400 now carries its two exits (upgrade litellm / responses()). Routing after the flip: chat protocol — bare names are OpenAI-compatible shorthand (wire form openai/<name>; OPENAI_API_KEY / OPENAI_BASE_URL still select the backend, and the missing key stays a build-time failure), litellm/ strips, openai/ opts out to the OpenAI SDK directly; responses protocol unchanged (OpenAI-SDK native, LiteLLM refused). The import cost is handled instead of dodged: local clients preload litellm on a background thread (first call then perceives 0.0s), and pageindex sets LITELLM_LOCAL_MODEL_COST_MAP=True via setdefault — LiteLLM's import otherwise blocks on a network fetch of its price map (fresh venv: 5.6s -> 1.3s; offline it hangs to the timeout). Also restores prompt_cache_key delivery, found dead during the flip's gating verification: openai-agents 0.20 no longer derives it from RunConfig.group_id, so both lanes sent nothing. ModelSettings.extra_body is the one channel all three model classes put on the wire (the bare kwarg is dropped by LiteLLM; extra_args[extra_body] collides with the responses model's own parameter — both wire-verified), and it is scoped to OpenAI destinations: LiteLLM plants extra_body as a literal field in other providers' bodies, and Anthropic rejects unknown fields — the anthropic wire test now pins the absence. Verified before landing: mock-server matrix (OPENAI_BASE_URL + bare name works through LiteLLM; gpt-named self-hosted models are NOT bridged off a custom base_url; prompt_cache_key on the wire in every OpenAI lane with distinct per-conversation keys; anthropic body clean) and live (sol answers through chat(), gpt-5.4 unchanged, responses() bare unchanged with the key on its wire).

…LLM's, verbatim Ray's ruling on the flip's remaining carve-out: a routing decision must never hide in a model-name prefix. openai/ now means what LiteLLM says it means (its openai provider), like every other name on the chat lane — the grammar is LiteLLM's with zero exceptions. The two defenses for keeping a direct carve-out had no concrete victim: debugging isolation (litellm is unavoidable in indexing anyway, and responses() IS the OpenAI-SDK-native door), and endpoint determinism (litellm sends chatcmpl for openai-provider models except the gpt-5 bridge, which never fires against a custom base_url — wire-verified). If a direct escape is ever needed, it will be a declared parameter, never name grammar. openai/-prefixed names keep the build-time OPENAI_API_KEY check for parity with bare names; responses() is untouched (bare and openai/ still drive the OpenAI SDK — LiteLLM cannot speak that protocol).

The prompt_cache_key delivery channel went through three iterations and settled on ModelSettings.extra_body; the _conversation_cache_key docstring still named extra_args from the middle iteration. The litellm install hint said >=1.30, below both our own pyproject floor (>=1.84.0) and the floor openai-agents' litellm extra declares (>=1.83). A user in a broken environment following it would land on a version the package itself rules out. The hint now matches the declared floor.

…e-key test to extra_body The config bundle hands its model string to the Agents SDK's own MultiProvider grammar, which refuses unknown prefixes (probe on 0.20: 'anthropic/x' -> UserError: Unknown prefix). _normalize_retrieve_model's litellm/ spelling is what keeps that door working — a link the existing self-referential assert (config["model"] == client.retrieve_model) could not catch. Pinned with a provider-slashed name. Also renames the cache-key delivery test to its real channel, extra_body — the extra_args name survived from the superseded delivery attempt.

…SDK's grammar _normalize_retrieve_model said what it does, not why. The litellm/ spelling exists because the Agents SDK resolves raw model strings with its own prefix grammar and refuses unknown prefixes — the name now points at that constraint.

…ile's siblings Every agents-dependent test in this file importorskips; without the guard this one errors where the others skip.

…acy fallback The documented surface becomes two role knobs: index_model builds the index, chat_model answers on the chat surfaces. model turns into the set-both umbrella (its 0.2.8 indexing semantics are a strict subset, so old configs run unchanged); summary_model and retrieve_model stay accepted as legacy role names. Resolution lives in ConfigLoader.load(), the one seam every consumer already passes through (client, CLI standard/md paths, flash's summary fallback, tree_optimize's default_model): new names win over old, specific over general, model sets every role, and code constants close each chain. The packaged yaml no longer ships model keys — key presence is what separates a user's explicit choice from a built-in default, and _validate_keys accepts the five model names explicitly. Consequences: with no config at all, classic-mode structure extraction now uses DEFAULT_INDEX_MODEL (gpt-5.6-luna) instead of the yaml's old gpt-4o-2024-11-20 line (ratified; flash-default users see no change). client.retrieve_model becomes a read-only alias for client.chat_model. The resolution matrix test pins one row per released generation: 0.2.8 (model), 0.3.0.dev (model+retrieve_model), 0.2.10.dev (all three legacy names), the new pair, umbrella-only, and mixed.

The CLI leads with --index-model; --model stays as its legacy synonym (the CLI only indexes, so the umbrella and the index role coincide). The flash branch's summary fallback gains the index position, and the standard branch now forwards --summary-model, which it had silently ignored — the flag's help always claimed it worked there. The md branch's unfiltered model=None no longer clobbers the default: the resolver treats None as unset.

Ray's pick for the out-of-box QA default; indexing stays on luna. sol runs tools on the Responses lane — current litellm bridges chat() there automatically; older litellm gets the guided 400 naming both exits.

The per-generation history lives in 7244ee4's message.

… thinking Each chat door gains its own protocol's native thinking control, forwarded verbatim with no invented vocabulary and no default of ours: chat_completions(reasoning_effort=...), responses(reasoning={...}), messages(thinking={...}). Unset sends nothing, so backend defaults (sol: medium, adaptive) are untouched. chat() stays answer-only. Delivery channels, each verified: the chat door rides extra_args["reasoning_effort"] — LiteLLM's own top-level kwarg on every supported openai-agents version, admitting non-enum values ("none"); newer openai-agents promotes it to the top-level argument and pops the duplicate. Wire-captured on a mock backend (/v1/chat/completions body carries it) and coexists with the Claude cache marker in one dict. The responses door rides ModelSettings.reasoning — coerced to the typed openai Reasoning object and forwarded verbatim by the Responses model; the envelope echoes the caller's dict. The messages door joins the existing anthropic passthrough dict, asserted through the real tool runner. LiteLLM semantics observed and accepted as-is: unknown models refuse the param loudly with LiteLLM's own remedies, and gpt-5.4+ names with an explicit effort route to /v1/responses even against a custom api_base (its documented pre-existing arm). The sol-class 400 guidance now names the third exit — an explicit effort routes on older litellm releases too. Cloud chat_completions rejects the new parameter like model/max_turns; responses()/messages() are local-only already.

The industry-standard per-request extension channel (openai/anthropic SDK trio): a dict merged verbatim into the backend request, last, so caller keys win over SDK-set ones. Routing per door: OpenAI-compatible destinations get a true body merge (ModelSettings.extra_body / the anthropic SDK's native extra_body); LiteLLM-routed providers take the keys as LiteLLM's own top-level kwargs instead, since LiteLLM plants extra_body as literal fields other providers reject. Cloud mode rejects it like the other local-only knobs. chat() stays answer-only.

1.97.0 is where the unset-effort chatcmpl->responses bridge landed (responses_api_bridge_check's on_constraint_enforcing_endpoint arm, A/B-verified against 1.96.2), so sol-class models work through the chat lane out of the box instead of 400ing until a manual upgrade. Three spots move together: the pyproject floor, the requirements.txt CI pin, and the install hint.

…esponses extra_body could not carry these: openai-agents' LitellmModel passes every ModelSettings sampling field as an explicit keyword and unpacks extra_args into the same call, so the common knobs collided with a bare TypeError on the LiteLLM lane (reproduced against a stub — Python call semantics, callee-independent). Named params ride ModelSettings fields, the one channel clean on every lane; responses() uses the protocol's own name (openai_responses maps ModelSettings.max_tokens to max_output_tokens on the wire) and the envelope now echoes the real value instead of a constant None. Both caps bound each backend call in the agent loop, not the whole run — documented. Cloud rejects them like the other local-only knobs; the long tail (frequency_penalty etc.) stays extra_body-blocked-loudly on that lane by choice.

The default chat_model is what put keyless users on the OpenAI lane, so the error now points at the knob that picks a different backend.

rejojer · 2026-08-16T16:42:06Z