🤖 feat: validated make update-models refresh + full catalog in Treat as by ibetitsmike · Pull Request #3930 · coder/xum

ibetitsmike · 2026-08-23T08:55:39Z

Summary

Adds a validated, idempotent make update-models flow that refreshes the vendored LiteLLM model catalog, and expands the custom-model "Treat as" dropdown to offer the full models.json catalog instead of only the ~21 curated KNOWN_MODELS.

Fixes#3727

Background

src/common/utils/tokens/models.json is a pruned vendor copy of LiteLLM's model_prices_and_context_window.json, but scripts/update_models.ts was manual-only and unwired, so the data had gone stale (last real refresh around #1103). Meanwhile the "Treat as" mapping (mappedToModel) is persisted as a plain string and resolved through resolveModelForMetadata -> getModelStats, which can already resolve any catalog entry, so restricting the dropdown to curated models was purely a UI limitation. Per the maintainer suggestion on the issue, the refresh is a make-process flag that fetches, validates pricing, and updates only when needed.

Implementation

Freshness:

scripts/update_models.ts is now a thin CLI over pure logic in src/common/utils/tokens/updateModelsData.ts (covered by bun test src): prune to retained fields, drop entries whose present cost fields are not finite non-negative numbers, then validate before writing (>= 500 chat entries as a truncation guard, <= 5% invalid-pricing drops, and every KNOWN_MODELS entry still resolvable via models.json plus models-extra). Validation also bounds shrinkage to 10% below the vendored catalog's chat-model count, curated coverage requires usable token limits (same bar as getModelStats) rather than key presence, and serialization sorts keys so upstream reordering never diffs. It writes only when the serialized content changed, so reruns are no-ops.
make update-models runs it; make build UPDATE_MODELS=1 makes the refresh a prerequisite of every catalog-consuming bundle, ordering it under parallel make and forcing those bundles stale (plain make build stays network-free).
.github/workflows/update-models.yml runs weekly (plus dispatch) and opens/updates a bot/update-models PR only when the data changed, reusing the auto-cleanup GitHub App token pattern so the bot PR triggers CI; the checkout is pinned to main and the job goes through make update-models.
knownModels.test.ts now reuses the same findMissingKnownModels coverage check instead of duplicating the key logic.
models.json refreshed in its own commit via the new target (2456 -> 3176 entries). Upstream deleted max_pdf_size_mb wholesale, so the capability test for that field now exercises the exported extraction function with injected metadata instead of a retired Gemini 1.5 fixture.

Treat as:

New listModelCatalogIds() (src/common/utils/tokens/modelCatalog.ts) returns canonical provider:model ids for every chat/responses catalog entry that resolves through getModelStats, unioned with the curated ids (~2300 ids on the refreshed data), normalized via normalizeToCanonical and deduped so gateway keys with direct-provider origins collapse into the canonical id whose metadata resolution actually uses. ModelsSection feeds it to the dropdown.
SearchableModelSelect caps rendering at 200 rows with a "+N more, keep typing to filter" footer so the popover stays responsive with the full catalog.

Validation

make update-models run twice: first refreshes, second prints "already up to date" with a clean tree.
Remote dogfood UAT (Coder Agents) on this exact head passed: full catalog + cap footer verified in the UI, non-curated mapping (openai:gpt-4o-mini) saves and inherits 128k context, "None (use own metadata)" clears it, curated Claude mappings still load; make update-models idempotency re-verified in the workspace; no console errors. Search latency measured at 4-31 ms per keystroke.
make lint-actions (actionlint + zizmor) clean for the new workflow.

Risks

The models.json refresh changes pricing/limits metadata app-wide (costs, context windows, capability gates). Curated models are pinned by models-extra overrides and the coverage validation, and bun test src passes on the refreshed data; the main residual risk is upstream metadata quality for non-curated models, which the pricing sanitization bounds.
The scheduled workflow only opens a PR (never pushes to main) and skips gracefully when app secrets are absent.

Generated with xum • Model: anthropic:claude-fable-5 • Thinking: xhigh

Upstream dropped max_pdf_size_mb entirely, so the capability inference test now exercises the extraction branch with injected metadata.

ibetitsmike · 2026-08-23T08:55:52Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:e84d314ea1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

- validate curated coverage by usable metadata, not key presence - bound catalog shrink against the vendored baseline - sort serialized keys to avoid reorder-only churn - normalize gateway catalog ids so rows match resolved metadata - order UPDATE_MODELS=1 refresh before catalog-consuming bundles - pin scheduled workflow checkout to main and route it via make

ibetitsmike · 2026-08-23T09:57:09Z

@codex review

Addressed all 8 findings in dda73bb (7 code fixes; the max_pdf_size_mb thread explains why we don't vendor the upstream-deleted values). Please take another look.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:dda73bb7b9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…fallback - capabilities now resolve with getModelStats' shared key preference so provider-scoped entries win in both lookups - models.json is a prerequisite of the main bundle so catalog-only refreshes rebuild it - update-models workflow references only canonical XUM_APP_* secrets

ibetitsmike · 2026-08-23T10:10:26Z

@codex review

Addressed all 3 round-2 findings in aa5110e: capabilities share getModelStats' lookup keys, models.json is a prerequisite of the main bundle, and the workflow uses only canonical XUM_APP_* secrets. Please take another look.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:aa5110ee78

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

- catalog rows must resolve stats from their own entry (resolveRawModelEntry), excluding provider-scoped duplicates shadowed by models-extra overrides - validation compares priced chat-model coverage against the vendored baseline so an upstream pricing-field rename cannot zero-price the catalog

ibetitsmike · 2026-08-23T10:24:35Z

@codex review

Addressed both round-3 findings in 4e04f5c: catalog rows must round-trip stats resolution to their own entry, and validation guards priced chat-model coverage against the vendored baseline. Please take another look.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:4e04f5cd7f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

- shrinkage measured over usable chat/responses entries (getModelStats bar), so token-limit loss and responses-mode loss both register - input- and output-priced chat coverage compared independently so renaming one cost field cannot hide behind the other

ibetitsmike · 2026-08-23T10:37:20Z

@codex review

Addressed both round-4 findings in 07dd882: usable chat/responses coverage (getModelStats' usability bar) drives the shrink guard, and input/output pricing coverage are tracked independently. Please take another look.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:07dd882b64

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ibetitsmike · 2026-08-23T10:49:33Z

@codex review

Addressed the round-5 finding in 673c0d8: priced coverage now spans chat and responses modes. Please take another look.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:673c0d8c8d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit:673c0d8c8d

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

- summarizeCatalog tracks usable/input-priced/output-priced coverage per mode plus total entries; every baseline mode is compared independently, closing pricing/mode-slice regressions the aggregate counters hid - baselines below a noise floor (20) skip relative checks so tiny modes cannot block refreshes forever - curated coverage reuses generateModelLookupKeys instead of the xai/moonshot special case

ibetitsmike · 2026-08-23T11:05:52Z

@codex review

Addressed all 3 round-6 findings in 430c0fb: per-mode coverage baselines (usable + per-field pricing, every mode plus total entries), and curated coverage via the shared runtime lookup keys. Please take another look.

Median input/output cost across mappable entries must stay within 100x of the vendored baseline, so a poisoned upstream cannot keep every field while scaling rates toward zero.

chatgpt-codex-connector · 2026-08-23T11:10:12Z

Codex Review: Didn't find any major issues. Swish!

Reviewed commit:430c0fb9ef

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ibetitsmike · 2026-08-23T11:10:18Z

@codex review

Addressed the security finding in 70578c3 (median price-magnitude guard vs the vendored baseline). Please review the current head.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:70578c3110

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit:70578c3110

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…ic magnitude baselines Replaces the bespoke inputPriced/outputPriced counters and cost-only medians with generic guards over the whole retained-field surface: - per-mode coverage counts for every retained field (pricing incl. image/cache/ tiered rates, token limits, capability flags; booleans count only when true) - positive-sample counts and medians for every numeric field across mappable modes, rejecting >100x median shifts including collapses to zero - per-mode entry counts so whole-mode omissions are caught directly

Catalog-wide counts and medians cannot see a targeted repricing of a single surviving row. validateModelData now takes the baseline catalog itself (simplifying callers) and rejects any surviving entry whose positive numeric field shifts more than 100x or vanishes.

ibetitsmike · 2026-08-23T11:35:17Z

@codex review

Addressed all 5 round-7 findings (4 review + 1 security) by generalizing the guards instead of adding more bespoke counters:

382ca0e: per-mode coverage baselines now span every retained field (all cost fields incl. image/cache/tiered rates, token limits, capability flags; booleans count only when true, so mass flips register as removal), and numeric magnitude medians span every numeric field (catching token-limit rescaling and zero-collapsed prices; the zero endpoint is no longer bypassed).
67ccf59: per-entry bounds against the vendored baseline catch targeted single-row shifts (scaling or deletion of a surviving row's positive numeric field beyond 100x fails validation).

Please review the current head 67ccf59.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:67ccf596a1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit:67ccf596a1

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…tion commits Raw-key per-entry comparison missed alias shadowing: a poisoned catalog can add or substitute a higher-precedence lookup key that hijacks runtime resolution while all counts and medians stay intact. Validation now compares, for every model id either catalog can serve, the entry each side resolves through getModelStats' own lookup preference, using runtime parseNum semantics (numeric strings compare by value, not as collapses) and requiring baseline-true capability flags to stay true per identity. Also adds the sanctioned automation attribution footer to the weekly bot commit/PR and documents the convention in the pull-requests skill.

ibetitsmike · 2026-08-23T11:56:58Z

@codex review

Addressed all 5 round-8 findings in c95ee88 by moving per-entry comparison from raw keys to runtime identities: for every model id either catalog can serve, both sides resolve their entry through getModelStats' own lookup preference and the resolved entries are compared. This covers alias-shadow additions and removed-key+alias substitutions with one mechanism, adds per-identity baseline-true capability-flag checks, and uses runtime parseNum semantics so numeric-string representation changes compare by value. The weekly bot commit/PR now carries the automation attribution footer, with the convention documented in the pull-requests skill.

Verified against live upstream: make update-models passes validation with the identity comparison enabled.

Please review the current head c95ee88.

chatgpt-codex-connector

💡 Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit:c95ee88162

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

ibetitsmike · 2026-08-23T12:12:25Z

@codex review

Re-requesting: the security review for head c95ee88 arrived, but the code review verdict for this head never landed. No code changes since the last request. The round-9 security finding (mode-flip repricing) was resolved with an explicit maintainer scope decision: the human-reviewed weekly bot PR is the trust boundary against a poisoned upstream; in-repo validation remains a best-effort sanity gate.

Please review the current head c95ee88.

chatgpt-codex-connector · 2026-08-23T12:17:08Z

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit:c95ee88162

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ration oversubscription) (coder#3939) ## Summary Fixes the three mechanisms that have been kicking fully validated PRs out of the merge queue: a unit-job Bun segfault caused by a desynced test-file exclusion list, integration-job cancellation caused by the job timeout colliding with jest's own test timeout, and integration test flakes caused by jest worker memory oversubscription on CI runners. ## Background Recent merge-queue evidence (all on green PRs whose diffs never touched the failing areas): 1. **Test / Unit segfault** (mq runs 32666269009, 32667158072, both kicked PR coder#3937): every test passes, then Bun 1.3.5 panics with `Segmentation fault` (exit 132). The logs show the crash is not teardown noise: it happens mid-`SandboxHostService` suite inside the monolithic coverage run. `src/node/services/sandbox/sandboxHostService.test.ts` was added to the `isolated_unit_tests` list in coder#3865 (QuickJS-heavy suites are known to crash Bun under coverage in a shared process) but was never added to the hand-maintained duplicate `find` exclusion list, so the suite ran a **second time** inside the shared process, where it segfaults intermittently. 2. **Test / Integration cancelled at the job cap** (mq runs 32632804824, 32639270365, 32640298486, kicking PRs coder#3931/coder#3930): the check-run annotation reads "The job has exceeded the maximum execution time of 10m0s". Healthy runs finish in ~4-5 min, but jest's integration `testTimeout` is 10 minutes (`jest.config.js`), so any single hung test makes jest wait right through the 10-minute job cap; the runner then kills the job and destroys all diagnostics. 3. **Recurring integration flakes** (`focus.test.ts` in run 32715740782 kicking PR coder#3932, `sendModeDropdown.test.ts` in run 32666269009, `anthropicCacheStrategy.test.ts` in run 32665306160, plus `analyticsEscape.test.ts` at 66s of its 60s budget in run 32719185534 kicking PR coder#3932's attempt 5, plus earlier `undo.test.ts`/`analyticsHeader.test.ts` kicks): all share one mechanism. CI runs jest with `--maxWorkers=100%`: 16 forked workers on the 16-core/64GB runner at ~4.7GiB peak each (per `scripts/lib/worker_budget.js`) oversubscribes memory (~75GiB peak) and thrashes. In the failing logs, suites that normally finish in seconds *pass* at 100-135s wall time while the only tests that *fail* are the ones with tight explicit timeouts (30s/45s), timing out in setup. The `--maxWorkers=100%` flag predates the repo's memory-aware worker budget (coder#1326 vs coder#3760), so CI never got the sizing fix that local runs already use. ## Implementation - **Unit job**: derive the `find` exclusions from the `isolated_unit_tests` array instead of maintaining a duplicate list, so the two can never drift again. This removes the double-run of the crashing suite (and fixes the whole desync class rather than one instance). No exit-code masking anywhere: a genuinely failing test still fails the job exactly as before. - **Integration job**: dropped the CLI `--maxWorkers` override so `jest.config.js`'s memory- and cgroup-aware `workerBudgetFor("jest")` sizes the pool (per Codex review; CLI flags take precedence over config), and `timeout-minutes: 10` → `20` so a hung test hits jest's 10-minute `testTimeout` and produces a real failure with logs instead of a log-destroying job cancellation. - **Worker recycling**: the first PR CI run under the bounded budget surfaced the complementary failure mode: integration suites leak memory across test files, so with fewer workers each one accumulates heap until it dies at V8's ~4GB cap (“Jest worker ran out of memory and crashed”, run 32720490232). `workerIdleMemoryLimit: "2GB"` recycles leaky workers between files. - **Tightest observed-killed test timeouts**: `focus.test.ts` 30s → 120s (aligned with its sibling `undo.test.ts`) `anthropicCacheStrategy.test.ts` 45s → 120s (peer provider tests already budget 45-150s per live call), and `analyticsEscape.test.ts` 60s → 120s after it died at 66s in run 32719185534. Other flagged tests already use the harness's 30s windows and should recover via the worker fix; they were left untouched to avoid churn. Live-API tests (e.g. `anthropicCacheStrategy`) are deliberately **not** excluded from `merge_group`: the mq run is the last gate that executes tests for the merged combination (main-push runs skip test jobs for the merge-queue bot), so quarantining them there would remove real coverage. The single observed kick was timeout-tuning, not provider outage. ## Validation - Red-green proof of the file-selection change: reproduced the old `find` invocation and the new derived one side by side; the old list contains `sandboxHostService.test.ts` (the double-run), the new list differs by exactly that one file (759 → 758, no other adds/removes). - `TEST_INTEGRATION=1 bun x jest tests/ui/review/focus.test.ts tests/ipc/providers/anthropicCacheStrategy.test.ts` passes locally (2/2, including the live Anthropic call). - `make lint-actions` (actionlint + zizmor) and `make static-check` pass. ## Risks CI-config only plus two test-timeout constants; no production code. The main tradeoff is failure-detection latency: a genuinely hung integration run is now killed at 20 min instead of 10 (but now with jest diagnostics at the 10-minute testTimeout), and the two touched tests take up to 120s before reporting a real hang. Suite wall time at 8 workers may shift slightly in either direction (less thrash, fewer lanes); healthy runs have ample headroom against the 20-minute cap either way. --- _Generated with `xum` • Model: `anthropic:claude-fable-5` • Thinking: `xhigh`_

ibetitsmike added 2 commits August 23, 2026 08:23

feat: validated make update-models flow + full Treat-as catalog (#3727)
6fa65fb

chore: refresh models.json from LiteLLM via make update-models
e84d314
Upstream dropped max_pdf_size_mb entirely, so the capability inference test now exercises the extraction branch with injected metadata.

This comment has been minimized.
Sign in to view