Conversation
Automated sync from stranske/Workflows Template hash: 108d8e23a769 Changes synced from sync-manifest.yml
🤖 Keepalive Loop StatusPR #272 | Agent: Codex | Iteration 0/5 Current State
🔍 Failure Classification| Error type | infrastructure | |
Keepalive Work Log (click to expand)
|
There was a problem hiding this comment.
Pull request overview
Syncs workflow templates from stranske/Workflows to enhance weekly agent-metrics reporting and add a bot-comment auth coverage preflight alongside existing terminal disposition coverage checks.
Changes:
- Adds
scripts/aggregate_agent_metrics.pyto aggregate downloaded NDJSON metrics into a markdown summary for the weekly workflow. - Extends weekly artifact selection to recognize bot-comment auth coverage artifacts and adds a new
bot_comment_auth_coverage.jspreflight. - Updates
agents-weekly-metrics.ymlto usesetup-node+setup-api-client, run the new preflight, upload its reports, and enforce both hard-block outcomes.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
scripts/aggregate_agent_metrics.py |
New Python aggregator that builds the weekly markdown summary from downloaded NDJSON metrics. |
.github/workflows/agents-weekly-metrics.yml |
Adds setup-node, bot-comment auth coverage preflight, uploads its outputs, and enforces both coverage hard-blocks. |
.github/scripts/weekly_metrics_artifacts.js |
Expands artifact-family classification to include bot-comment auth coverage artifacts via regex patterns. |
.github/scripts/bot_comment_auth_coverage.js |
New Node preflight script to summarize/enforce bot-comment App auth coverage with optional hard-blocking. |
| for index, entry in enumerate(entries): | ||
| is_terminal_disposition = entry.get("schema") == "workflows-terminal-disposition/v1" | ||
| if not is_terminal_disposition: | ||
| run_id = entry.get("run_id") or entry.get("workflow_run_id") | ||
| run_attempt = entry.get("run_attempt") | ||
| pr_number_for_key = _safe_int(entry.get("pr_number") or entry.get("pr")) | ||
| if run_id: | ||
| verifier_run_keys.add(f"run:{run_id}:attempt:{run_attempt or ''}") | ||
| elif pr_number_for_key is not None: | ||
| verifier_run_keys.add(f"pr:{pr_number_for_key}") | ||
| else: | ||
| verifier_run_keys.add(f"entry:{index}") | ||
|
|
||
| verdict = entry.get("verdict") | ||
| if verdict: | ||
| verdicts[str(verdict)] += 1 | ||
| if is_terminal_disposition: | ||
| terminal_records += 1 | ||
| disposition = entry.get("disposition") or entry.get("terminal_state") or "unknown" | ||
| terminal_dispositions[str(disposition)] += 1 | ||
| source_type = entry.get("source_type") or "unknown" | ||
| source_id = entry.get("source_id") or "unknown" | ||
| terminal_sources[f"{source_type}:{source_id}"] += 1 |
There was a problem hiding this comment.
_summarise_verifier only increments verifier_run_keys for non-terminal-disposition entries. If a dataset contains only (or additional) workflows-terminal-disposition/v1 records, runs will be under-counted (potentially 0) even though verifier activity exists. Consider adding run_id/run_attempt from terminal disposition records into the run key set when present (or tracking them separately) so the run count matches the records being summarized.
| def build_summary(entries: list[dict[str, Any]], errors: int) -> str: | ||
| buckets: dict[str, list[dict[str, Any]]] = { | ||
| "keepalive": [], | ||
| "autofix": [], | ||
| "verifier": [], | ||
| "terminal_disposition": [], | ||
| "autopilot": [], | ||
| "unknown": [], | ||
| } | ||
| timestamps: list[_dt.datetime] = [] | ||
|
|
||
| for entry in entries: | ||
| bucket = _classify_entry(entry) | ||
| buckets.setdefault(bucket, []).append(entry) | ||
| for key in ("timestamp", "created_at", "time", "run_started_at"): | ||
| ts = _parse_timestamp(entry.get(key)) | ||
| if ts is not None: | ||
| timestamps.append(ts) | ||
| break | ||
|
|
||
| keepalive = _summarise_keepalive(buckets["keepalive"]) | ||
| autofix = _summarise_autofix(buckets["autofix"]) | ||
| verifier = _summarise_verifier(buckets["verifier"] + buckets["terminal_disposition"]) | ||
| autopilot = _summarise_autopilot(buckets["autopilot"]) |
There was a problem hiding this comment.
This introduces a new aggregation script that drives a scheduled workflow, but there are no unit tests validating key behaviors (entry classification, timestamp parsing, verifier/terminal-disposition aggregation, and markdown output invariants). Adding a small test module under tests/ would help prevent regressions when metrics schemas evolve.
| function parseArgs(argv = process.argv.slice(2)) { | ||
| const options = { | ||
| dir: process.env.BOT_COMMENT_AUTH_COVERAGE_DIR || 'artifacts', | ||
| output: process.env.BOT_COMMENT_AUTH_COVERAGE_JSON || 'bot-comment-auth-coverage-summary.json', | ||
| markdown: process.env.BOT_COMMENT_AUTH_COVERAGE_MD || 'bot-comment-auth-coverage-summary.md', | ||
| artifact_selection_report: process.env.BOT_COMMENT_AUTH_ARTIFACT_SELECTION_JSON || '', | ||
| }; | ||
| for (let index = 0; index < argv.length; index += 1) { | ||
| const arg = argv[index]; | ||
| const next = argv[index + 1]; | ||
| if (arg === '--dir') { | ||
| options.dir = next; | ||
| index += 1; | ||
| } else if (arg === '--output') { | ||
| options.output = next; | ||
| index += 1; | ||
| } else if (arg === '--markdown') { | ||
| options.markdown = next; | ||
| index += 1; | ||
| } else if (arg === '--artifact-selection-report') { | ||
| options.artifact_selection_report = next; | ||
| index += 1; | ||
| } else if (arg === '--mode') { | ||
| options.enforcement_mode = next; | ||
| index += 1; | ||
| } else if (arg === '--hard-block-approved') { | ||
| options.hard_block_approved = next; | ||
| index += 1; | ||
| } | ||
| } | ||
| return options; | ||
| } | ||
|
|
||
| function main() { | ||
| const options = parseArgs(); | ||
| const files = collectJsonFiles(options.dir); | ||
| const readResult = readJsonRecords(files); | ||
| const report = summarizeBotCommentAuthCoverage(readResult.records, { | ||
| parse_errors: readResult.parse_errors, | ||
| read_errors: readResult.read_errors, | ||
| parsed_json_file_count: readResult.parsed_json_file_count, | ||
| non_auth_record_count: readResult.non_auth_record_count, | ||
| input_files: files, | ||
| input_file_count: readResult.file_count, | ||
| artifact_selection_report: readArtifactSelectionReport(options.artifact_selection_report), | ||
| enforcement_mode: options.enforcement_mode, | ||
| hard_block_approved: options.hard_block_approved, | ||
| }); | ||
| const markdownSummary = formatBotCommentAuthCoverageMarkdown(report); | ||
| fs.writeFileSync(options.output, `${JSON.stringify(report, null, 2)}\n`); | ||
| fs.writeFileSync(options.markdown, markdownSummary); | ||
| process.stdout.write(markdownSummary); | ||
| return report.status === 'fail' ? 1 : 0; | ||
| } | ||
|
|
||
| if (require.main === module) { | ||
| process.exitCode = main(); | ||
| } | ||
|
|
||
| module.exports = { | ||
| AUTH_SCHEMA, | ||
| COMPONENT_POLICIES, | ||
| COVERAGE_SCHEMA, | ||
| collectJsonFiles, | ||
| componentPolicy, | ||
| formatBotCommentAuthCoverageMarkdown, | ||
| isPotentialAuthCoverageFile, | ||
| normalizeArtifactSelectionSummary, | ||
| normalizePolicy, | ||
| parseArgs, | ||
| readArtifactSelectionReport, | ||
| readJsonRecords, | ||
| summarizeOrganicEvidence, | ||
| summarizeBotCommentAuthCoverage, | ||
| }; |
There was a problem hiding this comment.
This is a large new preflight script with a lot of normalization/policy logic (mode parsing, artifact selection parsing, organic evidence blockers, directory scanning) but no tests. Since the repo already has .github/scripts/tests/* coverage for other workflow scripts, consider adding focused tests for helpers like normalizeMode/normalizePolicy, isPotentialAuthCoverageFile, and summarizeBotCommentAuthCoverage to reduce risk of breaking scheduled workflows.
|
Closing as stale: newer sync workflow templates PR #339 is open for this repo. |
Sync Summary
Files Updated
Files Skipped
Review Checklist
Source: stranske/Workflows
Manifest:
.github/sync-manifest.yml