Skip to content

[Feature] mcode provider add should support context and output limits for custom/local providers #161

Description

@mrefaie

Problem

I configure a local llama.cpp server as a custom provider with:

mcode provider add --name my-provider --base-url https://example.com/v1 \
  --api-format openai-completions --model my-model \
  --api-key-env MCODE_PROVIDER_API_KEY --use

There is no CLI way to set the model's context window (ctx) or output limit at creation time.

This matters because model discovery (ModelDiscoveryClient in packages/local-runtime-v2/src/service/model-system/connectivity/discover-models.ts) only discovers modelId/displayName from GET /v1/models or /models — it does not probe num_ctx / n_ctx / context_length. At resolve time planCustomProviderResolution() in packages/local-runtime-v2/src/service/model-system/resolution/model-resolver-byok.ts falls back to BYOK_FALLBACK_MODEL_LIMITS (200_000 ctx / 16_384 output) when modelConfig.limit is unset. So a local model started with e.g. --ctx-size 32768 is silently resolved with the wrong window unless the user edits config.yaml manually.

Current chain

  • packages/tui/src/cli/program.ts (provider add): only --name, --base-url, --api-format, --model <id> (repeatable string[]), --api-key-env, --use.
  • packages/tui/src/cli/provider-command.ts (McodeProviderCliRequest): models: readonly string[], mapped as models.map((modelId) => ({ modelId })).
  • Downstream already supports limits: McodeProviderModelInput.limit?: { context?: number; output?: number } in packages/tui/src/provider/contract.ts, consumed by modelsFromInputs() / normalizeModelLimit() in packages/local-runtime-v2/src/service/model-system/management/service-input.ts and customProviderLimits() in model-resolver-byok.ts.

So this is a CLI-surface gap only.

Proposal

Extend mcode provider add with e.g.:

mcode provider add --name my-provider --base-url http://localhost:8080/v1 \
  --api-format openai-completions --model my-model \
  --api-key-env MCODE_PROVIDER_API_KEY \
  --context-limit 32768 --output-limit 4096 --use
  • Validate as positive safe integers, consistent with existing normalizeModelLimit() / updateLocalModelSelection() validation.
  • Document semantics for repeatable --model: simplest is "same limits apply to all listed models".
  • Consider whether provider update / existing updateUserModelParameters() path also needs CLI exposure, or if add-time support is sufficient for v1.
  • Update README.md + README_ZH.md + docs/examples.md (English primary per CONTRIBUTING.md).

Acceptance

  • mcode provider add ... --context-limit N --output-limit M persists custom_provider.<key>.models.<id>.limit = { context: N, output: M } in <dataDir>/config.yaml.
  • mcode provider list --json / provider view shows the configured limits.
  • Invalid (zero/negative/non-integer) values are rejected with a clear error.
  • Existing behavior without the new flags is unchanged.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions