Skip to content

docs: update coverage gap analysis to 94.03% #406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stranske merged 1 commit into from
Dec 31, 2025
Merged

docs: update coverage gap analysis to 94.03% #406

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 29 additions & 39 deletions docs/reference/coverage-gap-analysis.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,75 +1,65 @@
# Coverage Gap Analysis

> **Baseline**: 2025-12-31 (fix/codex-coverage-verification branch)
> **Overall Coverage**: 74.76%
> **Tests**: 592 passed, 0 skipped, 3 xfailed
> **Baseline**: 2025-12-31 (Updated after PR #401)
> **Overall Coverage**: 94.03%
> **Tests**: 715 passed, 3 xfailed, 1 failed

Copilot AI Dec 31, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test status format has changed from "592 passed, 0 skipped, 3 xfailed" to "715 passed, 3 xfailed, 1 failed". The new format includes "1 failed" test, which is concerning for a coverage report baseline. Typically, coverage baselines should be taken when all tests are passing (excluding expected failures marked as xfailed). Including a failed test in the baseline may indicate an unstable state. Please clarify whether this failed test is expected or if the baseline should be recalculated after fixing the failure.

Suggested change
> **Tests**: 715 passed, 3 xfailed, 1 failed
> **Tests**: 715 passed, 3 xfailed (no unexpected failures)

Copilot uses AI. Check for mistakes.
> **Target**: 95%

## Summary

| Metric | Value |
|--------|-------|
| Total Statements | 3,828 |
| Covered Statements | 2,946 |
| Missing Statements | 882 |
| Coverage | 74.76% |
| Gap to 95% | 20.24% (~775 statements) |
| Total Statements | 2,798 |
| Covered Statements | 2,631 |
| Missing Statements | 167 |
| Coverage | 94.03% |
| Gap to 95% | 0.97% (~27 statements) |

Copilot AI Dec 31, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The gap calculation appears to use an approximation symbol () when the value is exact. With 2,798 total statements and a target of 95%, exactly 2,658 statements need to be covered (0.95 × 2,798 = 2,658.1, rounded down to 2,658). With 2,631 currently covered, the gap is exactly 27 statements (2,658 - 2,631 = 27). The tilde () should be removed since this is a precise calculation, not an approximation. However, there's a more critical issue: the document shows 167 missing statements total, but claims only 27 statements are needed to reach 95%. This is contradictory - if 167 statements are missing, then covering 27 more would still leave 140 uncovered. Please verify and reconcile these conflicting numbers.

Suggested change
| Gap to 95% | 0.97% (~27 statements) |
| Gap to 95% | 0.97% (27 additional covered statements needed to reach 95%; 167 total uncovered) |

Copilot uses AI. Check for mistakes.

## Scripts by Coverage (Lowest to Highest)

### Tier 1: Zero Coverage (3 scripts, 234 statements)
### Tier 3: Medium Coverage 50-75% (3 scripts, 128 statements missing)

| Script | Statements | Coverage | Missing |
|--------|------------|----------|---------|
| `sync_tool_versions.py` | 74 | 0.00% | 74 |
| `update_residual_history.py` | 25 | 0.00% | 25 |
| `validate_version_pins.py` | 135 | 0.00% | 135 |
| `workflow_health_check.py` | 77 | 63.64% | 28 |
| `ledger_validate.py` | 205 | 69.27% | 63 |
| `classify_test_failures.py` | 123 | 69.92% | 37 |

### Tier 2: Very Low Coverage <50% (3 scripts, 248 statements missing)
### Tier 4: High Coverage 75-95% (3 scripts, 28 statements missing)

| Script | Statements | Coverage | Missing |
|--------|------------|----------|---------|
| `sync_test_dependencies.py` | 163 | 15.32% | 128 |
| `keepalive_metrics_collector.py` | 108 | 46.48% | 56 |
| `auto_type_hygiene.py` | 139 | 48.79% | 64 |
| `mypy_return_autofix.py` | 89 | 87.64% | 11 |
| `ledger_migrate_base.py` | 134 | 90.30% | 13 |
| `workflow_validator.py` | 72 | 94.44% | 4 |

### Tier 3: Medium Coverage 50-75% (5 scripts, 196 statements missing)

| Script | Statements | Coverage | Missing |
|--------|------------|----------|---------|
| `keepalive_metrics_dashboard.py` | 94 | 56.67% | 40 |
| `workflow_health_check.py` | 77 | 62.62% | 28 |
| `classify_test_failures.py` | 123 | 62.87% | 37 |
| `mypy_autofix.py` | 45 | 63.08% | 13 |
| `ledger_validate.py` | 205 | 65.32% | 63 |

### Tier 4: High Coverage 75-95% (7 scripts, 46 statements missing)

| Script | Statements | Coverage | Missing |
|--------|------------|----------|---------|
| `mypy_return_autofix.py` | 89 | 82.55% | 11 |
| `ledger_migrate_base.py` | 134 | 85.48% | 13 |
| `ci_failure_analyzer.py` | 108 | 87.35% | 11 |
| `fix_cosmetic_aggregate.py` | 20 | 92.31% | 1 |
| `coverage_history_append.py` | 53 | 92.75% | 2 |
| `workflow_validator.py` | 72 | 93.27% | 4 |
| `update_autofix_expectations.py` | 37 | 93.88% | 1 |

### Tier 5: At Target ≥95% (11 scripts)
### Tier 5: At Target ≥95% (23 scripts)

Copilot AI Dec 31, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Tier 5 section header claims "23 scripts" but only 22 scripts are listed in the table below. Please add the missing script or correct the count in the header. The listed scripts are: auto_type_hygiene.py, ci_coverage_delta.py, ci_history.py, ci_metrics.py, keepalive_metrics_dashboard.py, metrics_format_utils.py, sync_status_file_ignores.py, sync_test_dependencies.py, sync_tool_versions.py, update_residual_history.py, validate_version_pins.py, ci_cosmetic_repair.py, ci_failure_analyzer.py, fix_numpy_asserts.py, keepalive_metrics_collector.py, mypy_autofix.py, aggregate_agent_metrics.py, build_autofix_pr_comment.py, generate_residual_trend.py, pr_metrics_tracker.py, coverage_history_append.py, fix_cosmetic_aggregate.py, and update_autofix_expectations.py.

Suggested change
### Tier 5: At Target ≥95% (23 scripts)
### Tier 5: At Target ≥95% (22 scripts)

Copilot uses AI. Check for mistakes.

| Script | Statements | Coverage |
|--------|------------|----------|
| `auto_type_hygiene.py` | 139 | 100.00% |
| `ci_coverage_delta.py` | 61 | 100.00% |
| `ci_history.py` | 77 | 100.00% |
| `ci_metrics.py` | 120 | 100.00% |
| `keepalive_metrics_dashboard.py` | 94 | 100.00% |
| `metrics_format_utils.py` | 16 | 100.00% |
| `sync_status_file_ignores.py` | 94 | 100.00% |
| `sync_test_dependencies.py` | 163 | 98.30% |
| `sync_tool_versions.py` | 74 | 100.00% |
| `update_residual_history.py` | 25 | 100.00% |
| `validate_version_pins.py` | 135 | 99.51% |
| `ci_cosmetic_repair.py` | 256 | 99.71% |
| `ci_failure_analyzer.py` | 108 | 99.40% |
| `fix_numpy_asserts.py` | 38 | 98.15% |
| `keepalive_metrics_collector.py` | 108 | 99.30% |
| `mypy_autofix.py` | 45 | 98.46% |
| `aggregate_agent_metrics.py` | 201 | 97.23% |
| `build_autofix_pr_comment.py` | 105 | 97.04% |
| `generate_residual_trend.py` | 61 | 96.55% |
| `pr_metrics_tracker.py` | 68 | 95.65% |
| `coverage_history_append.py` | 53 | 95.65% |
| `fix_cosmetic_aggregate.py` | 20 | 95.00% |
| `update_autofix_expectations.py` | 37 | 97.30% |

## Verification Command

Expand Down
Loading