Skip to content

fix: thinking strength now follows the chosen level on Kimi/StepFun/MiniMax coding plans - #2067

Merged
Astro-Han merged 6 commits into
mainfrom
fix/kimi-thinking-effort
Aug 4, 2026
Merged

fix: thinking strength now follows the chosen level on Kimi/StepFun/MiniMax coding plans#2067
Astro-Han merged 6 commits into
mainfrom
fix/kimi-thinking-effort

Conversation

@Astro-Han

@Astro-HanAstro-Han commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Problem

Kimi Coding Plan did not expose thinking-strength controls. The root cause was a pre-#2050 orphan: kimi-coding-plan was missing from the PROVIDERS map in scripts/sync-model-metadata.mjs, so its metadata stayed frozen in a hand-written static block from #1164 (k3 pinned to a single max effort, k3-256k absent), and the runtime wire hard-coded max without ever reading the chosen level.

A systematic audit found two more instances of the same class:

  • stepfun-step-plan — same orphan shape (upstream segment existed, sync map missing, hand-written static block expanding the stepfun API segment, modelsDevId mislabeled as stepfun).
  • minimax-coding-plan — alias debt (declared modelsDevId: minimax and reached metadata through a static alias to the MiniMax API segment; the two models.dev segments could drift apart silently).

Changes (5 commits)

  1. kimi-coding-plan (f2fdc7b): sync map gains kimi-coding-plan → kimi-for-coding; the snapshot now declares k3 (toggle + low/high/max) and k3-256k (low/high/max); the hand-written static block is deleted; buildProviderOptions passes the chosen level through both wires (anthropic effort and openai-chat reasoningEffort), defaulting to max when unset; k3-256k joins the K3 adaptive-thinking wire.

  2. stepfun-step-plan (d61e178): sync map gains its own models.dev segment; the static block expanding stepfun records is deleted; modelsDevId corrected from the mislabeled stepfun to its own segment, and the import-time model guard now checks the stepfun-step-plan segment.

  3. minimax-coding-plan + sync contract (3a98f74): sync map gains its own segment (7 models, MiniMax-M3 toggle preserved); the alias line is deleted; registry modelsDevId and the fallback guard point at its own segment. Adds models-dev-sync-contract.test.ts and declares modelsDevId for every snapshot-backed provider (anthropic, openai, google, deepseek, kimi-coding-plan, zai-coding-plan, MiniMax, MiniMax-cn, gemini-cli, claude-subscription, openai-codex).

  4. Three-way sync contract (edef7eb): sync script emits the complete models.dev directory (GENERATED_MODELS_DEV_DIRECTORY); the contract gains a directory direction (registry base-URL host matching a directory provider must declare it); kimi off is explicitly rejected in the wire; fallback guards and intent comments added.

  5. External review round (Claude Opus + Codex) (b3f8471): both independent reviews found no P0/P1 but converged on real gaps, all fixed:

    • kimi off rejection was dead code (the entry gate normalized off to undefined before the case guard, silently turning an explicit off into max; a test even locked that in). The case now checks the raw thinkingLevel argument and rejects off with empty options.
    • the sync contract could not see alias-type orphans: a provider declaring a neighbour segment id (the exact pre-fix stepfun/minimax shape) passed all checks, and claude-subscription → openai passed too. Replaced host-based matching with a deterministic completeness check: every provider must have its own segment, a declared alias (xai-oauth/opencode-free/claude-subscription/openai-codex), or a whitelist reason (9 local/user-configured/volcengine entries). Mutation-tested: both bug shapes now fail the suite.
    • main() in the sync script had zero coverage (the place the orphan bug lived): added an end-to-end fixture test over the real main() path (9/9 script tests).
    • the wire-contract sweep only exercised kimi's anthropic branch: added an explicit openai-chat sweep for kimi-coding-plan.
    • GENERATED_MODELS_DEV_DIRECTORY shrank to { api?: string } (name was never consumed); dropped the unused FACTS_BY_PROVIDER alias.
    • stepfun guard now covers all 4 fallback models (the old slice(0, 3) was stale); minimax guard label says MiniMax Coding Plan.

Verification

  • core 772/772 pass (incl. the sync contract suite; mutation tests confirm the kimi, alias-mislabel, and neighbour-segment bug shapes all fail)
  • runtime 3089 pass, 4 pre-existing macOS /var/private/var symlink failures (identical on main, unrelated)
  • sync script tests 9/9; biome lint / format:check / full-workspace typecheck clean
  • Behavior: k3/k3-256k → [low, high, max]; step-3.7-flash → [low, medium, high]; MiniMax-M3 toggle-only, no regression
  • StepFun official docs cross-checked: step-router-v1 is a routing model (reasoning: false is correct; the old hand-written true was wrong), and the low/medium/high effort levels match the snapshot

Confirmed external facts

  • Kimi Coding Plan API accepts low/high/max for k3 / k3-256k (verified by the user; no live-check script needed). The snapshot's effort levels are safe to pass through; no STATIC narrowing back to ['max'].

Deferred (tracked, not blocking)

  • kimi protocol-decision consolidation onto resolveModelRuntime + provider-options camelCase cleanup (runtime refactor, suggested as a separate PR)
  • kimi-for-coding sends effort: 'max' despite having no declared reasoning_options (pre-existing behavior, not a regression; models.dev declares none, the UI offers no menu, and default-max is the sensible wire)
  • openrouter maxOutputTokens drift for 4 models (snapshot refresh, noted in the generated diff)
  • zai-coding-plan glm-4.6 and tencent-token-plan 8/9 models still lack metadata (upstream segments do not cover them; would need local pins + live checks)

Kimi Coding Plan was a pre-#2050 orphan: it was missing from the
sync-model-metadata PROVIDERS map, so its metadata stayed a hand-written
static block (k3 pinned to a single 'max' effort, k3-256k absent), and
the runtime wire hard-coded max and never read the level.
- sync-model-metadata.mjs maps kimi-coding-plan to models.dev's
kimi-for-coding provider; the snapshot now declares k3 (toggle +
low/high/max) and k3-256k (low/high/max), and the stale static block
is deleted (regenerated snapshot also carries a little openrouter
drift from upstream)
- buildProviderOptions passes the chosen level through both Kimi wires
(anthropic effort and openai-chat reasoningEffort), defaulting to max
when unset; k3-256k joins the K3 adaptive-thinking wire
- the thinking wire contract test now sweeps kimi-coding-plan's declared
levels automatically, closing the silent-forget gap for this provider
Live-check pending: confirm Kimi's coding API accepts low/high for
k3/k3-256k before relying on non-max levels.
stepfun-step-plan was a dormant twin of the kimi-coding-plan orphan: the
sync map only covered stepfun / stepfun-ai / stepfun-ai-step-plan, so
stepfun-step-plan lived on a hand-written static block that expanded
stepfun (API) records and pinned reasoning facts by hand.
- sync-model-metadata.mjs maps stepfun-step-plan to its own models.dev
provider; the snapshot now declares all four Step Plan models with
upstream effort levels (step-3.7-flash low/medium/high, the 3.5-flash
pair low/high) and the static block is deleted
- provider-registry.ts: stepfun-step-plan modelsDevId was mislabeled as
stepfun; now resolves its own provider facts and the model guard
checks the stepfun-step-plan snapshot segment
- note: step-router-v1's reasoning capability now follows models.dev
(false) instead of the hand-written true; no thinking options were
declared either way, so the wire behavior is unchanged
Declared levels are swept by the existing thinking wire contract, so
this closes the same silent-forget gap for stepfun-step-plan.
…d guard the sync map
The MiniMax Coding Plan access path was the last alias-style provider:
it declared modelsDevId minimax and reached metadata through a static
alias to the MiniMax API segment, so the two models.dev segments could
drift apart without anyone noticing (their docsUrl already differs).
- sync-model-metadata.mjs maps minimax-coding-plan to its own models.dev
provider; the snapshot now carries a dedicated segment (7 models,
MiniMax-M3 toggle preserved) and the alias line is deleted
- provider-registry.ts: minimax-coding-plan modelsDevId resolves its own
segment facts and the fallback model guard checks its own segment
- new models-dev-sync-contract test closes the silent-forget gap for
good: every registry modelsDevId must resolve to the provider's own
snapshot segment (or alias metadata), and every snapshot segment must
be declared by the registry with its own id — adding a provider to the
sync map without wiring its registry entry is now a test failure
- registry modelsDevId now declared for every snapshot-backed provider:
anthropic, openai, google, deepseek, kimi-coding-plan, zai-coding-plan,
MiniMax, MiniMax-cn, gemini-cli (aliases xai-oauth/opencode-free keep
their existing alias ids)
…nc contract
Independent review of the previous fix found the sync contract only
covered the already-declared universe: a registry provider with no
modelsDevId and no sync-map entry (the exact kimi-coding-plan bug shape)
was invisible to both directions, and the forward check was tautological
(every modelsDevId was assigned from the same generated facts object it
compared against).
- sync-model-metadata.mjs now also emits GENERATED_MODELS_DEV_DIRECTORY:
the complete models.dev provider catalog (id, name, api), not just the
mapped subset
- models-dev-sync-contract.test.ts becomes three-way:
* forward: a declared modelsDevId must name a provider that exists in
the directory (catches sync keys pointing at nonexistent sources)
* reverse: every snapshot segment must be declared by the registry
(unchanged)
* directory: a registry provider whose base URL host matches a
directory provider must declare modelsDevId — the kimi bug shape is
now a test failure. Localhost/self-hosted endpoints are excluded
- registry: kimi-coding-plan fallback models now guarded at import time
like stepfun/minimax; claude-subscription and openai-codex declare
their anthropic/openai segment sources; kimi-for-coding comment notes
the intentionally-missing knob is not a sync gap
- runtime: kimi case explicitly rejects off (future-proofing for a
models.dev 'none' declaration) instead of the dead level !== 'off'
guard; openai-chat namespace comment documents the ai-sdk camelCase
alias dependency; highspeed branch comment states the no-wire intent
- tests: openai-chat high passthrough asserted, kimi off entry-gate
behavior asserted, step-router-v1 no-variants intent locked
Verified the new contract has teeth: removing kimi's modelsDevId makes
both the reverse and directory checks fail.
…ract, main() coverage
Two independent external reviews (Claude Opus, Codex) both found no
P0/P1, but converged on real gaps in the previous review round:
- kimi 'off' rejection was dead code: the entry gate normalized off to
undefined before the case guard, so an explicit off silently became
max (the test even locked that in). The case now checks the raw
thinkingLevel argument and rejects off with empty options.
- the sync contract could not see alias-type orphans: a provider
declaring a *neighbour* segment id (the exact pre-fix stepfun/minimax
shape) passed all three checks, and claude-subscription pointing at
openai passed too. Replaced host-based matching with a deterministic
completeness check: every provider must have its own segment, a
declared alias (xai-oauth/opencode-free/claude-subscription/openai-
codex), or a whitelist reason (9 local/user-configured/volcengine
entries). Both mutation shapes now fail the suite.
- sync-model-metadata.mjs main() had zero coverage (the place the
orphan bug lived): added an end-to-end fixture test over the real
main() path, and exported PROVIDERS/main(argv) to make it testable.
- the wire-contract sweep only exercised kimi's anthropic branch: added
an explicit openai-chat sweep for kimi-coding-plan.
- GENERATED_MODELS_DEV_DIRECTORY shrank to { api?: string } (name was
never consumed); dropped the unused FACTS_BY_PROVIDER alias.
- stepfun guard now covers all 4 fallback models (slice(0,3) was stale);
minimax guard label says MiniMax Coding Plan.
Deferred (recorded, not implemented): k3 low/high live check against
the Kimi API (B1), kimi protocol-decision consolidation onto
resolveModelRuntime (B3) and camelCase namespace cleanup (D3) as a
runtime refactor, kimi-for-coding forced-max semantics pending the live
check, user-visible upstream drift noted in the PR description.
Verified: core 772/772, runtime 3089 pass / 4 pre-existing macOS symlink
failures, sync scripts 9/9, lint/format/typecheck clean. Mutation tests:
claude-subscription→openai and minimax-coding-plan→MiniMax.id both now
fail the contract suite.
@Astro-Han

Copy link
Copy Markdown
ContributorAuthor

Review round (edef7eb) — internal subagent review

An internal multi-angle review found three gaps in the initial fix; all addressed in this commit:

  1. The sync contract test was hollow (H1/P2): it only covered the already-declared universe, so the exact kimi bug shape (registry entry exists, no modelsDevId, no sync-map entry) was invisible to both directions, and the forward check was tautological (every modelsDevId was assigned from the same generated facts object it compared against). → The sync script now emits the complete models.dev directory (GENERATED_MODELS_DEV_DIRECTORY, 178 providers); forward became a directory-existence check; a new directory direction requires any registry provider whose base-URL host matches a directory provider to declare modelsDevId. Discrimination verified: removing kimi's modelsDevId turns both reverse and directory checks red.
  2. kimi off silently upgraded to max (M3/P4): the case guard was unreachable and the fallback would turn a future none declaration into max. → The case explicitly rejects off (empty options); tests lock the entry-gate semantics.
  3. openai-chat high never asserted (H2): → added, covering low/max/high/off on the openai-chat path.

Also: kimi fallback import-time guard (matching the stepfun/minimax convention), claude-subscription/openai-codex declare their source segments, highspeed-branch intent comment, camelCase alias dependency comment, step-router-v1 no-variants intent locked.

Verified: core 772/772 (the new directory check caught a real lm-studio false positive on its first run, fixed by excluding localhost), runtime 3089 pass / 4 pre-existing macOS symlink failures, scripts 8/8, lint/format/typecheck clean.

@Astro-Han

Copy link
Copy Markdown
ContributorAuthor

External review round (Claude Opus + Codex) — b3f8471

Two independent external reviews (Claude Opus, 55 turns; Codex) both found no P0/P1, but converged on three real gaps in the previous round; all fixed:

  1. kimi off rejection was dead code (Codex P2): the entry gate normalized off to undefined before the case guard, so an explicit off silently became max — a test even locked that behavior in. → The case now checks the raw thinkingLevel argument; off returns empty options. Tests updated: off → {} on both protocols, undefined → max preserved.
  2. The sync contract could not see alias-type orphans (both reviews P2; Codex proved by mutation that claude-subscription → openai passed all three checks). A provider declaring a neighbour segment id (the exact pre-fix stepfun/minimax shape) was indistinguishable from a legitimate alias. → Replaced host-based matching with a deterministic completeness check: every provider must have its own snapshot segment, a declared alias (xai-oauth / opencode-free / claude-subscription / openai-codex), or a whitelist reason (9 local / user-configured / volcengine entries). Mutation-verified: both the alias-mislabel shape and the neighbour-segment shape now fail the suite.
  3. main() in the sync script had zero coverage (Claude A3 — the place the orphan bug lived): → added an end-to-end fixture test over the real main() path (mapped provider generates a segment, unmapped provider appears only in the directory); exported PROVIDERS / main(argv) to make it testable.

Additional fixes: wire-contract sweep gained an explicit kimi openai-chat variant (Claude B2); GENERATED_MODELS_DEV_DIRECTORY shrank to { api?: string } (name was never consumed, Codex P3); dropped the unused FACTS_BY_PROVIDER alias; stepfun guard now covers all 4 fallback models (the slice(0, 3) was stale, Claude C1); minimax guard label says MiniMax Coding Plan (C2).

Deferred with reasons: k3 low/high live check → now confirmed closed (the Kimi API accepts low/high/max; see the follow-up comment), protocol-decision consolidation onto resolveModelRuntime + camelCase namespace cleanup (runtime refactor, separate PR), kimi-for-coding forced max (pre-existing, UI-unreachable).

Verified: core 772/772, runtime 3077 pass / 4 pre-existing macOS symlink failures, scripts 9/9, lint/format/typecheck clean. StepFun official docs cross-checked: step-router-v1 is a routing model (reasoning: false correct).

@Astro-Han

Copy link
Copy Markdown
ContributorAuthor

B1 confirmed closed — external fact verified

The Kimi Coding Plan API accepts low / high / max thinking-effort levels for k3 / k3-256k (verified by the user; no live-check script needed). This means:

  • k3's ['low', 'high', 'max'] levels from the models.dev snapshot are safe to pass through — no STATIC narrowing back to ['max'];
  • all "Live-check pending" notes are resolved; there is no runtime-400 risk on non-max levels;
  • the extra live-check script is unnecessary.

Remaining deferred items (recorded in the review rounds, not blocking merge):

  • kimi protocol-decision consolidation onto resolveModelRuntime + provider-options camelCase cleanup (runtime refactor, suggested as a separate PR)
  • kimi-for-coding sends effort: 'max' with no declared reasoning_options (pre-existing behavior, not a regression; models.dev declares none, UI offers no menu, default-max is the sensible wire)
  • openrouter maxOutputTokens drift for 4 models: noted in the generated snapshot diff, updates naturally on the next sync

@Astro-Han
Astro-Han marked this pull request as ready for review August 4, 2026 05:22
@Astro-Han
Astro-Han merged commit d67063e into mainAug 4, 2026
11 checks passed
@Astro-Han
Astro-Han deleted the fix/kimi-thinking-effort branch August 4, 2026 05:22
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@Astro-Han