Conversation
Automated sync from stranske/Workflows Template hash: f0578787979a Changes synced from sync-manifest.yml
🤖 Keepalive Loop StatusPR #284 | Agent: Codex | Iteration 0/5 Current State
🔍 Failure Classification| Error type | infrastructure | |
Keepalive Work Log (click to expand)
|
There was a problem hiding this comment.
Pull request overview
Syncs workflow templates and supporting scripts to improve weekly agent metrics reporting and coverage/preflight observability across workflows.
Changes:
- Extend
aggregate_agent_metrics.pyto emit a JSON “summary contract”, enrich verifier/model stats, and include detailed parse-error reporting. - Add/propagate new telemetry and reporting fields across weekly metrics artifact selection, terminal disposition coverage, and bot-comment auth coverage.
- Update workflows to upload new JSON artifacts and emit wrapper terminal-disposition records for bot-comment handler runs.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
scripts/aggregate_agent_metrics.py |
Adds parse-error detail tracking, verifier model metadata reporting, and emits a JSON summary contract alongside markdown. |
.github/workflows/agents-weekly-metrics.yml |
Publishes the new JSON weekly metrics summary artifact. |
.github/workflows/agents-bot-comment-handler.yml |
Adds skip_reason output and emits/upload wrapper terminal disposition telemetry. |
.github/workflows/agents-81-gate-followups.yml |
Adjusts jq invocation to compact output. |
.github/scripts/weekly_metrics_artifacts.js |
Reports missing/selected priority families and exposes priority family status details. |
.github/scripts/terminal_disposition_coverage.js |
Adds verifier model compatibility checks and richer artifact-selection/status normalization for terminal disposition coverage. |
.github/scripts/terminal_disposition.js |
Extends terminal disposition schema normalization with model/reason/verifier_mode fields. |
.github/scripts/coverage_monitor_summary.js |
New script to aggregate coverage monitor reports into a single JSON/MD contract. |
.github/scripts/bot_comment_auth_coverage.js |
Tracks wrapper decisions about reusable invocation and improves organic evidence requirement reporting. |
|
|
||
| - name: Upload wrapper terminal disposition | ||
| if: always() | ||
| uses: actions/upload-artifact@v7 |
There was a problem hiding this comment.
actions/upload-artifact is referenced with the floating @v7 tag here, while the rest of the repo pins it to a commit SHA (e.g. @043fb46... # v7). Please pin this action to a specific SHA to match repo convention and avoid supply-chain risk.
| uses: actions/upload-artifact@v7 | |
| uses: actions/upload-artifact@043fb46c9edb0b0c7c6e8f6f0b0d3f0a4f7a6c52 # v7 |
| model_text = str(model) | ||
| verifier_models[model_text] += 1 | ||
| if model_text.lower() in unsupported_models: | ||
| unsupported_verifier_models[model_text] += 1 |
There was a problem hiding this comment.
Model names are matched against unsupported_models using model_text.lower(), but the counters use the original model_text as the key. This can split counts across different casings (and also make the unsupported model counters not line up cleanly with the normalized unsupported list). Consider normalizing the counter keys consistently (e.g., lowercasing once) before incrementing.
| model_text = str(model) | |
| verifier_models[model_text] += 1 | |
| if model_text.lower() in unsupported_models: | |
| unsupported_verifier_models[model_text] += 1 | |
| normalized_model_text = str(model).lower() | |
| verifier_models[normalized_model_text] += 1 | |
| if normalized_model_text in unsupported_models: | |
| unsupported_verifier_models[normalized_model_text] += 1 |
| "by_artifact_family": dict(sorted(family_counts.items())), | ||
| "by_artifact": dict(sorted(artifact_counts.items())), | ||
| "by_reason": dict(sorted(reason_counts.items())), | ||
| "details": [detail.as_dict() for detail in parse_error_details], | ||
| } |
There was a problem hiding this comment.
_parse_error_contract includes a details array with an entry per parse error. For a badly corrupted or very large input, this can make the JSON summary artifact extremely large. Consider bounding the exported details (similar to the markdown table cap) and/or adding an env flag to include full details only when needed, while keeping the aggregate counts always.
Sync Summary
Files Updated
Files Skipped
Review Checklist
Source: stranske/Workflows
Manifest:
.github/sync-manifest.yml