fix: preserve reviewed model fallback - #2779
Conversation
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
Important Review skippedAuto reviews are limited based on label configuration. 🚫 Excluded labels (none allowed) (7)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Workflow source neededPR #2779 needs either a linked GitHub issue or one valid non-issue Workflow Source before PR metadata automation can manage it safely. Please do one of:
Once a valid source is present, this warning will not be reposted. |
Automated Status SummaryHead SHA: 259f005
Coverage Overview
Coverage Trend
Top Coverage Hotspots (lowest coverage)
Low Coverage Files (<50.0%)
Updated automatically; will refresh on subsequent CI/Docker completions. Keepalive checklistScopeNo scope information available Tasks
Acceptance criteria
|
There was a problem hiding this comment.
Pull request overview
This PR addresses regressions in reviewed-model selection and fallback behavior surfaced during the current consumer sync wave, ensuring the system continues to use reviewed defaults when configured slots/models are unusable.
Changes:
- Preserve reviewed default slot fallbacks when a slot config file exists but yields zero usable slots.
- Route CI failure triage and GitHub Models provider selection through the reviewed model registry, and improve related diagnostics.
- Apply configurable policy confidence levels to Wilson interval calculations in the model benchmark evaluator (with validation + tests).
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/llm_registry.py | Falls back to reviewed default slots when slot entries resolve to no usable models. |
| tools/llm_provider.py | Improves GitHub Models client diagnostics and updates failure behavior when no reviewed model is available. |
| tools/evaluate_model_benchmark.py | Adds confidence-level-driven z-score computation and passes it into Wilson intervals. |
| tools/ci_failure_triage.py | Uses reviewed registry selection for provider models instead of a default constant. |
| tests/tools/test_llm_registry_selection.py | Updates/extends tests for slot-config fallback behavior. |
| tests/tools/test_evaluate_model_benchmark.py | Adds confidence-level policy coverage and validation test. |
| templates/consumer-repo/tools/llm_registry.py | Mirrors reviewed-slot fallback behavior for consumer templates. |
| templates/consumer-repo/tools/llm_provider.py | Mirrors GitHub Models warning behavior for consumer templates. |
| templates/consumer-repo/tools/ci_failure_triage.py | Mirrors reviewed-registry model resolution for consumer templates. |
| client = self._get_client() | ||
| if not client: | ||
| raise RuntimeError("LangChain OpenAI not available") | ||
| raise RuntimeError("GitHub Models client unavailable or no reviewed model is configured") |
|
|
||
| model_name = _configured_langchain_model("github-models", fallback=DEFAULT_MODEL) | ||
| if not model_name: | ||
| logger.warning("No reviewed GitHub Models selection is configured") |
|
|
||
| model_name = _configured_langchain_model("github-models", fallback=DEFAULT_MODEL) | ||
| if not model_name: | ||
| logger.warning("No reviewed GitHub Models selection is configured") |
| def test_all_unusable_slot_entries_fall_back_to_reviewed_defaults( | ||
| monkeypatch: pytest.MonkeyPatch, tmp_path: Path | ||
| ) -> None: | ||
| registry_path = tmp_path / "registry.json" | ||
| slots_path = tmp_path / "slots.json" | ||
| _write_registry(registry_path) | ||
| _write_slots(slots_path, profile="misspelled-profile") | ||
| monkeypatch.setenv(registry.ENV_MODEL_REGISTRY_CONFIG, str(registry_path)) | ||
| monkeypatch.setenv(registry.ENV_SLOT_CONFIG, str(slots_path)) | ||
|
|
||
| assert registry.load_slot_config()[0].model == "model-balanced" |
|
Runner dispatch state for autofix on PR #2779. Do not edit. |
|
Runner dispatch state for codex on PR #2779. Do not edit. |
|
Autofix updated these files:
|
Fixes review-backed regressions surfaced by the current consumer sync wave.\n\n- falls back to reviewed slots when configured entries are all unusable\n- resolves CI-triage models through the reviewed registry\n- clarifies missing GitHub Models configuration\n- applies the policy confidence level to Wilson intervals\n\nValidation: focused pytest suite (113 passed), template sync/completeness, and diff check.