Uh oh!
There was an error while loading. Please reload this page.
feat(opencode): model capability tiers for small/local models - #44242
Open
yanglinfang wants to merge 37 commits into
Open
feat(opencode): model capability tiers for small/local models#44242yanglinfang wants to merge 37 commits into
yanglinfang wants to merge 37 commits into
Conversation
Add a three-tier capability taxonomy (minimal/default/vendor) resolved per model: explicit per-model config tier, then the models.dev model-tier catalog field (read defensively until upstream anomalyco#41372 lands), then a parameter-count heuristic over the model id, with a vendor family guard so frontier ids stay untouched. Band edges are exported constants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Route minimal-tier models to a new compact task-first prompt (prompt/minimal.txt,
mode A/B decision, real-tool-call and step-budget rules) and default-tier models
to a trimmed default (prompt/default-compact.txt). The vendor family ladder is
unchanged and byte-identical for claude/gpt/gemini/kimi ids. A per-model config
field `prompt` replaces the family prompt entirely; file loading uses the
existing {file:./path} config substitution resolved against the config dir.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>Minimal-tier models get only the core roster (bash, read, write, edit, glob, grep, todowrite, plus the internal invalid fallback); default tier keeps the standard roster minus apply_patch, with the gpt usePatch swap forced off so edit/write survive. The tier is resolved once in session tools resolution and passed into the registry; per-agent permission/tool overrides still filter later in request prep. Untiered callers keep the existing behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add sanitizeGrammarSafeSchema: sanitizeOpenAISchema plus $ref/$defs inlining (cycle-safe), anyOf/oneOf/allOf flattening to the first non-null variant, boolean-only additionalProperties, and typed single-object items — the subset llama.cpp's GBNF converter accepts. ProviderTransform.schema applies it when the resolved tier is minimal or default, which covers every non-vendor @ai-sdk/openai-compatible model while keeping claude/gpt/gemini/kimi schemas byte-identical (kimi keeps its moonshot sanitizer even over openai-compatible). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add per-model config sampling { temperature, topP, topK } consulted before the
substring ladders in ProviderTransform. Minimal tier pins the llama.cpp launch
tuning (0.1 / 0.95 / 20) ahead of the ladders; default tier uses those values
only when the ladder has no entry, and only when the tier is backed by explicit
config or a parameter-count match, so unknown cloud ids (deepseek-v4-flash)
and frontier families keep today's values. Config-declared options such as
chat_template_kwargs and reasoning already reach the openai-compatible request
body via the providerOptions namespace routing; a test now locks that in.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>Replace the fixed 20k compaction buffer with a window-proportional reserve: min(20_000, max(2_048, floor(context * 0.15))). The explicit compaction.reserved config keeps absolute priority and large explicit windows keep the 20k reserve via the min(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An unset/zero model context limit no longer disables proactive compaction. Unless compaction.auto is explicitly false, overflow math assumes a conservative 32k usable window and logs a warning once per session. On a provider ContextOverflowError with no configured limit, the failing request's estimated input size is recorded as a session- level cap upper bound (in-memory per session) that shrinks the default window for subsequent overflow checks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The auto-continue message after an overflow compaction only blames media attachments when media parts were actually dropped with the compacted head. Text-only overflows now state the real cause and carry an explicit negative instruction so small models do not repeat the attachment story to the user. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(a) maxOutputTokens: when limit.output is unset, fall back to min(32_000, max(1_024, floor(limit.context * 0.25))) instead of a flat 32k that could swallow an entire small window. Unset limit.context keeps the flat fallback (the overflow layer's conservative default is not double-applied here). (b) LLMRequestPrep.prepare clamps maxOutputTokens to max(256, usable_window - estimated_input) reusing the overflow math. prepare is the chosen seam because the fully composed request (system, messages, resolved tools) first exists there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified by trace that PATCH /config already re-resolves provider model limits without restart: ConfigHttpApi.update marks the instance for disposal, disposeMiddleware runs InstanceStore.dispose after the response, and runDisposers(directory) invalidates every InstanceState cache including the Provider state that holds resolved limits. No code change needed; this adds a focused test pinning the per-directory disposer invalidation that path relies on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New config compaction.prompt replaces the built-in summary template
used when compacting a session. The value goes through the standard
config {file:./path} substitution so the prompt can live in a file
(e.g. a simpler template for minimal-tier models). Plugin-provided
compaction prompts keep precedence; default behavior is unchanged.
Also regenerates the legacy JS SDK types, which picks up the Wave 1
per-model tier/prompt/sampling config fields alongside compaction.prompt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>When the prompt loop reaches its last permitted step, the request now carries no tools and toolChoice "none" instead of only the MAX_STEPS_PROMPT prose claiming tools are disabled. The lastStep flag flows from the loop through LLM.StreamInput into request prep, where the resolved roster is emptied (StructuredOutput survives so json_schema turns can still deliver their result). The V2 runner already drops tools on the last step, so only the V1 engine changes. Applies to all tiers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sort Glob.scan results lexicographically before mapping directory plugins to file URLs, making numeric filename prefixes (00_, 05_, 10_) a real load-order convention instead of a decorative one. Load order previously depended on filesystem enumeration order. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Split trigger hooks into two classes: blocking hooks (tool.execute.before, permission.ask, command.execute.before) keep first-throw propagation because a throw is semantic, while all accumulating hooks (chat.params, chat.headers, chat.message, event, experimental transforms, tool.execute.after, ...) now catch per-plugin errors, log them, and continue with the remaining plugins. Previously the first throwing hook cancelled every later plugin's hook for the turn. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On the minimal tier, a doom_loop detection that would raise a permission ask instead strips the offending tool: the third identical call fails with exact recovery text and the tool is excluded from the session's next two prepared requests (resolveTools). The public permission Action enum stays untouched — extending it would break the generated SDK/config contract — so the behavior is tier-gated: default and vendor tiers keep today's ask semantics. Strip state is per-session in-memory, mirroring overflow.ts's learned limits; small model calls (title/summary) do not consume the strip budget. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
experimental_repairToolCall now attempts mechanical repair of the arguments before routing to the invalid tool: smart quotes to straight, single-quoted keys/strings to double where unambiguous, python True/False/None literals, trailing commas, and unbalanced brackets — ported from a production router-side repair table as a pure helper. The repaired call is returned only when it parses and validates against the tool's schema; otherwise the existing lowercase-name fix and invalid fallback apply unchanged. Saves a full provider round-trip per malformed call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GET /session/:id/context-budget on the V1 instance API reports the session's context arithmetic for arithmetic routing: effective model limits and tier, compaction reserve, usable window, dry-run baseline cost (system prompt, tool roster, instructions), history estimate with provider-reported usage from the last finished assistant message, and projected next-request input/headroom. The dry run mirrors the prompt loop's model/agent resolution and request prep assembly without dispatch side effects (documented approximations in the schema). Regenerated SDK artifacts (openapi.json, hey-api client). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every provider request now carries x-opencode-est-input-tokens, -history-tokens, -baseline-tokens, -tools-tokens, -limit-context, -limit-output, -usable, -tier, -session-id, -agent, and -subagents headers, computed from what request prep already assembled after tool resolution (reusing the C6 clamp estimates, no double estimation). est-input = history + baseline; baseline includes the tools figure. The subagent roster is computed in the prompt loop from the same source as the task tool description and threaded through StreamInput. Native values precede the chat.headers plugin hook so plugins may override but can never silently lose them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Typed bus events (SSE-visible via the existing bridge) for context
arithmetic decisions: session.overflow.detected {tokens, usable,
reserve, action} at both overflow gates (prompt-loop check via
SessionCompaction.isOverflow and the processor step-finish check),
session.compaction.started/completed {before_tokens, after_tokens}
around compaction processing (after_tokens approximates the retained
tail), session.output.clamped {requested, granted} when the C6 clamp
reduces the output budget (prepare reports it, llm.ts publishes), and
session.tool.stripped {tool} augmenting B4's strip log. Regenerated
SDK artifacts for the extended event union.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>The context-budget response's baseline.tools now carries tools_detail
[{id, chars, est_tokens}] — per-tool serialized cost (id + description
+ provider-transformed JSON schema). Adds the C3 CI budget gate:
minimal-tier baseline (system_prompt + tools) must stay <= 6k tokens
(measured 5189: prompt 1598 + tools 3591 across 8), default tier
<= 12k (measured 6669: prompt 1347 + tools 5322 across 12). Regenerated
SDK artifacts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>Session status disagreed with on-disk artifacts in both directions
(idle-with-nothing, error-after-successful-writes). Additive fixes,
existing status semantics untouched: (a) session.error events now carry
parts_written when the erroring turn had already completed file-writing
tool parts (write/edit/apply_patch), computed across the turn's
assistant messages; (b) every runLoop exit path (break, error, abort)
emits session.turn.completed {sessionID, status: idle|error,
parts_written, last_error?} via Effect.onExit as the reconciliation
channel for graders and routers. Regenerated SDK artifacts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>Token.estimate(text, hint?) resolves a bytes-per-token density from a format tag or filename extension (csv/tsv 1.3, json/ndjson/jsonl 1.5, log 2.0, prose default 4 unchanged). Token.register(fn) installs a custom estimator that wins over the density table (an embedder wires a exact tokenizer through it via plugin). Compaction prune passes the read filename as a hint; general estimates keep the default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When a tool call fails validation, the repair callback retries with top-level argument keys converted snake_case -> camelCase (file_path -> filePath, old_string -> oldString; generic transform, nested keys are left alone as they may be data). The transformed args are accepted only when they validate against the tool schema. Retires the downstream plugin 05_file_path_arg_rename. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A task_id without the ses session-id prefix on a fresh dispatch is a model-invented label, not a resume; SessionID.make previously died on it and burned the whole step (anomalyco#1367). The label is now ignored and the dispatch proceeds fresh. camelCase variants (taskId/taskID) are dropped by the parameter schema decode, which the new tests pin. Retires the downstream product plugin 07_task_id_fresh_dispatch_strip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
experimental.omit_model_identity omits the model identity line (You are powered by the model named ...) from the system prompt environment block. Default off everywhere except the minimal tier, which omits the line by default as part of the C3 baseline budget; an explicit false restores it. Retires the downstream plugin 03_strip_model_id_prompt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
getSmallModel gains a substring fallback (flash/nano/haiku/mini over family and id) after the exact-family ladder misses, so config-defined models with an empty family can still be auto-selected. A configured small_model that does not resolve now logs a warning naming the missing model instead of failing silently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reasoning-in-text models leak <think>...</think> blocks into normal assistant turns; the only scrub today is title generation. On the minimal tier the completed text part is now scrubbed with the same regex at text-end (streaming deltas still flow raw). Other tiers are byte-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The volatile date line in the environment block invalidated the prompt cache prefix daily (observed 83% -> 27% hit-rate collapse). On the minimal/default tiers SystemPrompt.environment drops the date from the env block and request prep appends it as a trailing system message, so the leading system message is byte-stable across days. Vendor tiers and small utility calls are byte-identical to upstream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified current defaults: no chunkTimeout is applied anywhere unless provider config sets one (only openai gets a headerTimeout default), so slow local decode had no stall guard and a hung stream waited forever. Minimal/default-tier models on @ai-sdk/openai-compatible endpoints now default chunkTimeout to 300_000 ms when the provider config sets no explicit chunkTimeout/timeout. Scoped to openai-compatible so cloud SDK behavior is untouched; applied before the SDK cache key so tiered and vendor models on one provider get distinct SDK instances. Verified by trace through the existing streaming suite (the wrapSSE chunk guard is now exercised by every openai-compatible llm test). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stream middleware beside the existing wrapLanguageModel transform,
gated on capabilities.toolcall === false or the minimal tier, finally
consuming the stored toolcall capability on the request path. Detects
<tool_call>{json}</tool_call> blocks, fenced json blocks, and bare
JSON objects of the {name|tool, arguments|parameters|input} shape,
validates the name against the prepared tools, converts to a native
tool-call stream part and suppresses the source text, rewriting a stop
finish to tool-calls so the step loop executes the tool. Conservative:
text is held only while it can still become a call and passes through
untouched on any ambiguity; capable models bypass entirely. Detection
patterns ported from a downstream router's prose-call lifting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>…olution
SessionTier.vendor() and the parameter-count heuristic read model.api.id
unconditionally; config-defined models (and the compaction suite's fake
model) can carry no upstream api id, so the E5 text-end scrub crashed the
processor mid-stream ("undefined is not an object (evaluating
'id.includes')") and left compaction summaries marked errored, breaking
anchored re-compaction. Fall back to the opencode model id (then empty).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>…op semantics
Follow-up wave to the tier/limits series, driven by an end-to-end validation
run of this branch against a 4B local model on a 116-task agent benchmark.
Two findings from that run shaped it:
- The minimal tool roster is protective, not merely small. A patch restoring
`task` to it cost a research-task category 0.868 -> 0.396: the model
delegated work it should have done directly, then exhausted its step budget.
A prompt-level fix scored worse (0.354). The roster stays narrow by default.
- The failure mode moved rather than disappeared. Seven of eight remaining
failures were timeouts, not overflow; one turn ran 70 steps / 1.83M tokens
before the provider gave up. `steps` alone does not bound a turn.
W6-1 config-owned tier roster + custom-tool exemption
MINIMAL_TIER_TOOLS becomes a default, overridable per model via
`tier_tools: { include, exclude }`. Tools contributed by plugins or a
`tool/` directory are exempt from the tier cut by default: the cut exists
to trim opencode's own surface, and silently dropping an integrator's tool
turns an advertised capability into a no-op. `exclude` still wins over the
exemption; `invalid` is not removable, being the LLM layer's landing slot
for malformed tool calls.
W6-2 enforced subagent enum on task
`subagent_type` is published as a schema enum of the agents this agent may
actually reach, from the same list the description renders (extracted as
`permittedSubagents` so the two cannot drift). Listing them only in prose
leaves the field free-form, and a small model will invent a name or reach
for an agent it is not permitted to use. No-ops when the permitted set is
empty, since an empty enum would reject every call.
W6-3 token and wall-clock turn budgets
New `turn_tokens` / `turn_seconds` agent config, both routed through the
existing B1 structural stop rather than adding a second termination path:
whichever trips first, the next request carries the max-steps directive and
no tools. Token accounting sums per-request usage the way isOverflow does.
W6-4 headroom-aware reads
Request preparation publishes the budget arithmetic it already computes;
`Tool.Context.budget()` exposes live remaining headroom, and `read` refuses
a slice that would not fit, reporting the numbers and a concrete narrower
call. Proactive compaction structurally cannot cover this: a single
oversized tool result goes from under-budget to over-window inside one step
and never crosses the trigger on the way up (observed at 57,632 tokens
against a 56,320 window).
W6-5 honest task failure semantics
A subagent stopped by a budget returns `state="max_steps"` with partial
output and parent-directed recovery text, instead of leaking the raw stop
directive. Models routinely answer that directive by restating it, so the
subagent's final text was often the directive itself, which reads to the
parent as an instruction addressed to it and gets surfaced as an
unexplained failure even when real work was done.
W6-6 deliverable contract
Optional `expected_artifacts` on a prompt. If the turn would go idle
without them, the agent is told once, precisely what is missing. Exactly
one nudge, and none at all when a budget already forced the text-only step
and the tools are gone.
Also: `script/build.ts` gains `--target <name>` to build a single named
target regardless of host platform (`--single` is host-only, so producing a
Linux container binary from a Windows box meant a full 12-target run).# Conflicts: # packages/opencode/src/tool/task.ts
…am semantics Upstream removed Qwen sampling defaults (anomalyco#43310) so serving-stack defaults win; the heuristic default tier no longer pins TIER_SAMPLING — only a config-declared tier does. Minimal tier is unchanged. Upstream getSmallModel intentionally ignores models without family metadata (anomalyco#33926); drop the id-substring fallback and keep the missing-small_model warning. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
The following comment was made by an LLM, it may be inaccurate: Based on my search, I found one potentially related PR: Related PR Found:
Other search results show related work on context management and small model support (like PR #39397 on truncation handling for smaller LLMs and PR #43713 on per-model compaction config), but PR #44242 is the current PR and should not be marked as a duplicate of itself. The PR #11377 is worth reviewing to ensure no overlapping scope, though it appears to address a different aspect of model tier handling. |
Enough1122
commented
Aug 23, 2026
Diff too large for automated batch review (67 changed files, +6588/-124) — recommending manual human review. |
- one overflow evaluation primitive (tokenTotal/reserveFor/evaluate) shared by the compaction gate, step-finish gate, turn budget, and context-budget endpoint; the step-finish gate now honors outputTokenMax like compaction - ToolRegistry.permittedSubagents exposed and reused by the telemetry header and both task-tool consumers (computed once per tools() build) - shared countFileWrites and THINK_BLOCK_RE between processor and prompt - one vendor-family ladder (model-family.ts) consumed by SystemPrompt and SessionTier, replacing two keep-in-sync copies - context-budget endpoint reuses SessionPrompt.currentModel and SessionCompaction.estimate instead of reimplementing both - max-steps stop markers derived from MAX_STEPS_PROMPT itself - drop the unused Token.register estimator hook; memoize tier detection; hoist the textcall middleware gate; fix banned star imports; reuse the shared test model fixture across seven test files Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Closes#41372
Reason: current OpenCode does not support small model with limited context window, like Qwen 4B. It will continuously trigger compaction. This pr introduce a minimal system prompt to handle it gracefully.
Type of change
What does this PR do?
Small local models (~4B) currently get the frontier-tuned system prompt, the full tool roster, and schema features (
format/pattern/anyOf/$ref) their grammar-constrained servers can't handle — on a 32k window the fixed 20k compaction reserve plus a ~28–34k baseline leaves almost no usable context.This adds a per-model
tier(minimal|default, from config or a parameter-count heuristic). Minimal tier gets a compact ~1.1k-token prompt, a reduced tool roster, grammar-safe schemas, and pinned sampling. It also makes the context arithmetic honest (proportional compaction reserve, window-aware output caps, format-aware token estimates), enforces max-steps/doom-loop stops structurally instead of by prose, lifts text-shaped tool calls into native ones for models that can't emit them, and addsGET /session/:id/context-budgetfor debugging budgets.Vendor models are untouched: regression tests assert their prompts, tools, and schemas stay byte-identical. Where this branch overlapped recent upstream decisions it yields to them — sampling stays unset for heuristic-sized models (#43310) and
getSmallModelstill ignores models without family metadata (#33926); pinning only applies when the tier is declared in config or the model sizes to minimal.How did you verify your code works?
packages/opencode(3446 pass) andpackages/core(1098 pass) suites green on currentdev, typecheck green in every touched package.To reproduce locally: serve any ≤9B model behind an OpenAI-compatible endpoint (e.g.
ollama pull qwen3:4b), point a provider at it inopencode.json, and checkGET /session/:id/context-budget— or settier: "minimal"explicitly on any model.Screenshots / recordings
Other tests
context-budget for qwen3:4b — tier resolves to minimal, proportional reserve 4915 on a 32k window, ~5.1k request baseline
same model, tier "default" vs "minimal" — the config knob trims the roster from 12 to 8 tools and the baseline from 6.6k to 5.1k tokens
Checklist