Refactor Copilot Review Instructions and Skills - #799
Conversation
There was a problem hiding this comment.
Pull request overview
Refactors the GitHub Copilot review bootstrap to route through a provider-independent code-review skill, and updates scripts/pr_review.py + tests to accept a stable machine-readable coverage marker and to block “unstated coverage” with a dedicated exit code.
Changes:
- Introduce a new
code-reviewskill (in both.agents/skillsand the Claude plugin distribution) that standardizes review procedure and the structured coverage marker. - Slim
.github/copilot-instructions.mddown to a bootstrap + output contract while preserving the repository’s disproved-claims ledger. - Extend
scripts/pr_review.pyand its tests to parse<!-- fleet-review: ... -->and to return exit 45 when coverage is unstated.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| scripts/tests/test_pr_review.py | Updates corpus/status tests to cover the new structured marker and new exit-45 behavior. |
| scripts/pr_review.py | Adds marker parsing and blocks unstated coverage with exit 45. |
| AGENTS.md | Routes “reviewing a change set” to the new code-review skill. |
| .github/copilot-instructions.md | Reduces to review bootstrap/runbook + preserves disproved-claims ledger. |
| .agents/skills/code-review/SKILL.md | Adds provider-independent review skill and defines the structured coverage marker contract. |
| .agents/skills/pr-review-conduct/SKILL.md | Updates “mechanics live elsewhere” guidance to point to scripts/pr_review.py + bootstrap. |
| .agents/skills/copilot-instructions-keeper/SKILL.md | Adjusts description to reflect bootstrap role of Copilot instructions. |
| .claude-plugin/fleet-skills/skills/code-review/SKILL.md | Distributes the new code-review skill into the Claude plugin skill set. |
| .claude-plugin/fleet-skills/skills/pr-review-conduct/SKILL.md | Mirrors the same “mechanics live elsewhere” update for the Claude plugin skill set. |
| .claude-plugin/fleet-skills/skills/copilot-instructions-keeper/SKILL.md | Mirrors the bootstrap-description update for the Claude plugin skill set. |
| .claude-plugin/fleet-skills/.source-digest | Updates the generated skill-set digest. |
| .claude-plugin/fleet-skills/.claude-plugin/plugin.json | Registers the new code-review skill in the plugin manifest. |
Suppressed comments (1)
scripts/tests/test_pr_review.py:911
- The docstring for this test still says that treating a no-coverage body as a failure would “cry wolf”, but
scripts/pr_review.py statusnow blocks oncoverage=unstatedwith exit 45. Update the docstring to reflect that this body shape is recognized ascoverage=unstated(not UNVETTED), but it is still a blocking outcome forstatusbecause it provides no proof of full diff coverage.
def test_a_round_stating_no_coverage_at_all_is_unstated(self) -> None:
"""28 of the 332 bodies are an overview and a change list, and that shape is current.
It interleaves with the counted one throughout rather than preceding it, and one pull
request carries both across its two rounds, so failing on it would cry wolf on about one
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Uh oh!
There was an error while loading. Please reload this page.
ptr727
commented
Aug 17, 2026
Copilot review round 4955381803 reported Suppressed comments (1).
Fixed in f1c24d7. The test now states that this is a recognized |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 53 out of 53 changed files in this pull request and generated no new comments.
Suppressed comments (2)
scripts/tests/test_pr_review.py:2844
- This contract test reads the marker from the
.agents/skills/copy, but the runbook explicitly points Copilot at.github/skills/code-review/SKILL.md. Reading the generated.github/skills/file here would better validate the actual Copilot-facing contract (and would fail if the generated tree wasn’t regenerated/committed).
.github/skills/code-review/SKILL.md:17 - The GitHub Copilot skill lives under
.github/skills/, but this step tells the reviewer to load skills from.agents/skills/. In downstream repos (where.agents/skills/is not carried), this instruction is impossible to follow and will prevent the intended routing to the per-language/workflow skills.
3. Load every applicable skill from `.agents/skills/`:
ptr727
commented
Aug 18, 2026
Suppressed finding responses Review round on
Review round on
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 53 out of 53 changed files in this pull request and generated no new comments.
Suppressed comments (1)
scripts/build_dist.py:180
is_stale()only considers directory entries when validating.github/skills/(and the plugin'sskills/), so a stray file at the distribution root (for example an accidental hand-addedREADME.mdor.DS_Store) would not be detected as drift andbuild_dist.py --checkcould still pass. Since the docs and help text describe--checkas failing whenever a generated distribution differs from.agents/skills/, it should also treat any non-directory entry under the distribution root as stale (and ideally require the root directory to exist even when there are zero skills).
ptr727
commented
Aug 18, 2026
Review round on
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 53 out of 53 changed files in this pull request and generated no new comments.
Suppressed comments (1)
scripts/pr_review.py:19
- The
statussubcommand docstring says exit 0 implies “the round covering the head read the whole diff”, butreport_verdict()returns 0 early when there are no head-covering reviews (if not head_reviews(pr): return 0). This makes exit 0 ambiguous (no review yet vs full coverage), so the docstring should describe the no-review case explicitly.
ptr727
commented
Aug 18, 2026
Review round on
|
Uh oh!
There was an error while loading. Please reload this page.
Summary
code-reviewskill that routes changed files to the applicable language, documentation, and workflow skillspr_review.pyparse the marker and block unstated coverage with exit 45Root Cause
The Copilot instruction file duplicated provider-independent rules and embedded API mechanics already implemented by
scripts/pr_review.py. Copilot review output also had no stable machine-readable coverage contract, while an unstated coverage result did not produce a blocking exit code.Validation
uvx ruff@latest check .uvx ruff@latest format --check .uvx mypy@latestOPERATIONS.md(79% total coverage, 97% forscripts/pr_review.py)python3 scripts/build_dist.py --checkpython3 scripts/repo_gate.pyFollow-Up
Closes#793