Skip to content

feat(scripts): sync the full models.dev fact set (pricing, description, knowledge, input limits) - #2399

Merged
Astro-Han merged 7 commits into
apache:mainfrom
Thinkya1:feat/2329-models-dev-sync
Aug 8, 2026
Merged

feat(scripts): sync the full models.dev fact set (pricing, description, knowledge, input limits)#2399
Astro-Han merged 7 commits into
apache:mainfrom
Thinkya1:feat/2329-models-dev-sync

Conversation

@Thinkya1

Copy link
Copy Markdown
Contributor

Summary

Synchronizes the models.dev model fact set into the core catalog and Runtime pricing snapshots.

  • Extend model facts with descriptions, lifecycle, input limits, knowledge cutoff, structured output, modalities, and refresh timestamps.
  • Generate builtin pricing from models.dev while preserving access-path-specific local supplements.
  • Prefer a model's provider-visible input limit when resolving context capacity.
  • Surface model descriptions and knowledge cutoffs in the model picker.
  • Keep the generated metadata and pricing snapshots committed because they are imported by the build/runtime paths.

Why

Issue #2329 tracks replacing the hand-maintained builtin pricing snapshot with models.dev-backed facts. Keeping the sync in one script makes pricing and model metadata reproducible while preserving explicit local exceptions for providers that need access-path-specific rates.

Verification

  • node --test scripts/sync-model-metadata.test.mjs — 11 passed.
  • npm --workspace @maka/core run test — 805 passed.
  • New Runtime pricing/context contract tests — 5 passed.
  • Runtime TypeScript build — passed.
  • git diff --check — passed.

The full Runtime suite was also run; unrelated existing model-factory-tool-call-index tests fail against the current AI SDK baseline. UI and Desktop typechecks are currently blocked by the latest upstream/mainconversationKey / Astryx ChatLayoutProps mismatch, outside this PR's files.

Closes#2329

@Thinkya1
Thinkya1 marked this pull request as ready for review August 7, 2026 06:59
@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks for extending the existing metadata sync instead of adding another generation path. I found three issues around authority ordering and pricing semantics.

P1: Static inputLimit overrides the access-path context limit

resolveSelectedModelContextWindow() currently prefers generated inputLimit over both model and metadata contextWindow:

https://github.com/maka-agent/maka-agent/blob/d34f914cdb091d9c91a1290380f53708c6535f9e/packages/runtime/src/context-budget-policy.ts#L422-L427

For openai-codex/gpt-5.5 and gpt-5.6-sol, the generated metadata contains inputLimit: 922000, while the Codex access path limits those models to 272K and 372K. The resolver therefore returns 922K and delays compaction until the provider rejects the request.

The precedence needs to reflect the source of the fact. A user declaration or access-path/runtime limit should constrain static public metadata, not the other way around. A regression test with the two real Codex models would catch this.

P1: Rebasing can drop the relay user's declared context window

The PR currently conflicts with main. Current main gives an openai-compatible relay's user-declared contextWindow highest priority, but the PR-side implementation does not contain that branch.

When resolving the conflict, the ordering should remain:

  1. User declaration for the configured access path
  2. Access-path or runtime fact
  3. Static models.dev metadata

Otherwise a relay user's explicit limit can be replaced by a provider response or generated guess.

P2: Subscription providers are assigned API token prices

The generator includes providers such as GitHub Copilot and Gemini CLI:

https://github.com/maka-agent/maka-agent/blob/d34f914cdb091d9c91a1290380f53708c6535f9e/scripts/sync-model-metadata.mjs#L118-L124

For example, github-copilot:gpt-5.5 receives OpenAI's per-token API price even though the user is billed through a subscription or request quota. Maka will then show that amount as a dollar cost in Usage and session details.

Pricing should be included based on whether the access path is actually billed per token. Subscription and plan-based providers should remain unpriced unless there is a rate that describes the user's real billing path. A Copilot regression test would make that policy explicit.

@Astro-Han

Copy link
Copy Markdown
Contributor

P2: Do not apply the base rate to requests covered by a higher pricing tier

toPricing() drops context_over_200k and tiers, while PricingConfig and computeCost() retain only one unconditional rate. The generated base rate is therefore applied even when the request belongs to a more expensive tier.

This affects current mapped models. For example, models.dev lists google:gemini-2.5-pro at $1.25/$10 per 1M input/output tokens below 200K, but $2.50/$15 above 200K. Long-context calls will consequently be underreported in Usage. The new “keeps base rates and ignores tiered rates” test currently locks in that incorrect behavior.

Until the runtime pricing contract can select a tier from the request usage, the safe minimal behavior is to omit generated pricing for models with context-dependent rates. Alternatively, preserve the tier structure in PricingConfig and make computeCost() select the applicable rate.

中文

P2:不要把基础费率用于应按更高阶梯计价的请求

toPricing() 丢弃了 context_over_200ktiers,而 PricingConfigcomputeCost() 只保留并无条件使用一组费率。因此,即使请求已经进入更高价格阶梯,Usage 仍会按基础价计算。

当前映射模型中已经存在实际影响。例如 models.dev 中 google:gemini-2.5-pro 在 200K 以下的输入/输出价格为每百万 token 1.25/10 美元,超过 200K 后则为 2.50/15 美元。长上下文调用的成本因此会被低估。新增的 “keeps base rates and ignores tiered rates” 测试反而固化了这个错误语义。

在 runtime 价格契约能够根据请求用量选择阶梯之前,最安全的最小方案是不为含上下文分层价格的模型生成价格。另一种方案是让 PricingConfig 保留阶梯结构,并由 computeCost() 选择适用费率。

@Astro-Han
Astro-Han merged commit c2d0909 into apache:mainAug 8, 2026
12 checks passed
@Astro-Han

Copy link
Copy Markdown
Contributor

Thanks for the thorough work — approved and merged. A few non-blocking follow-ups we noticed:

  • Z.AI Coding Plan still carries three per-token rates in the local supplement, although this access path is quota-based. It may be clearer to leave these entries unpriced.
  • Flat pricing cannot accurately represent differentiated reasoning/audio rates or the tiered MiniMax CN rate. Omitting those entries until the pricing contract supports them would avoid misleading estimates.
  • The newly extended ModelInfo fields are not yet accepted by the strict runtime-policy codec. This is harmless with current fetchers, but the boundary should be aligned before live discovery emits them.
  • The new PDF modality helpers currently have no production consumer and could be deferred until attachment preflight uses them.

None of these affects the merged feature’s primary behavior.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(scripts): sync the full models.dev fact set (pricing, description, knowledge, input limits)

2 participants

@Thinkya1@Astro-Han