Add workflow telemetry state contracts - #1838
Merged
Merged
Conversation
stranske
had a problem deploying
to
agent-high-privilege
April 25, 2026 17:37 — with 
GitHub Actions
Error
Contributor
Contributor
Contributor
🤖 Keepalive Loop StatusPR #1838 | Agent: Codex | Iteration 0/5 Current State
🔍 Failure Classification| Error type | infrastructure | |
Contributor
Keepalive Work Log (click to expand)
|
stranske
temporarily deployed
to
agent-high-privilege
April 25, 2026 17:41 — with
GitHub Actions
Inactive
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 93b2264035
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+173
to
+175
| BOT_COMMENT_AUTH_REQUIRED_ORGANIC_EVENTS: ${{ vars.BOT_COMMENT_AUTH_REQUIRED_ORGANIC_EVENTS || 'pull_request,workflow_run' }} | ||
| BOT_COMMENT_AUTH_ORGANIC_COMPONENTS: ${{ vars.BOT_COMMENT_AUTH_ORGANIC_COMPONENTS || 'agents-bot-comment-handler-wrapper,reusable-bot-comment-handler' }} | ||
| BOT_COMMENT_AUTH_ORGANIC_EXPECTED_MODE: ${{ vars.BOT_COMMENT_AUTH_ORGANIC_EXPECTED_MODE || 'client-id' }} |
There was a problem hiding this comment.
These new organic-evidence defaults are only added in the repository-local workflow, but consumer sync does not use this file as the source for workflow updates. In maint-68-sync-consumer-repos.yml the workflows section copies from templates/consumer-repo/..., and templates/consumer-repo/.github/workflows/agents-weekly-metrics.yml still lacks these env vars, so synced consumer repos will keep required_organic_events unset and the new telemetry contract stays effectively disabled there. Mirror these env defaults into the template workflow (or the actual sync source) to avoid cross-repo behavior drift.
Useful? React with 👍 / 👎.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds new “telemetry state contracts” to make weekly metrics and the sync/dependabot campaign output more machine-readable and diagnosable, while keeping enforcement warning-only by default.
Changes:
- Extends bot-comment auth coverage reporting to include “organic evidence” requirements (per event/component) and surfaces blockers in the weekly metrics preflight.
- Persists sync/dependabot campaign state + a compact markdown run summary as artifacts on every run.
- Adds a validation contract to detect drift between retained campaign queue items and the stats derived from them.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
.github/scripts/bot_comment_auth_coverage.js |
Adds organic evidence parsing/summarization, integrates it into blockers, and exposes it in JSON/MD output. |
templates/consumer-repo/.github/scripts/bot_comment_auth_coverage.js |
Intended to mirror the bot-comment auth coverage updates for consumer templates. |
.github/workflows/agents-weekly-metrics.yml |
Wires new organic evidence env vars into the weekly metrics preflight run. |
.github/workflows/maint-82-sync-dependabot-campaign.yml |
Writes campaign state + summary files and uploads them as artifacts (always). |
.github/scripts/sync_dependabot_campaign.js |
Adds validation + markdown run summary generation, and stores validation in campaign state. |
.github/scripts/__tests__/sync_dependabot_campaign.test.js |
Adds unit tests for the new validation and markdown artifact summary. |
.github/scripts/__tests__/bot-comment-auth-coverage.test.js |
Adds tests ensuring missing/fulfilled organic evidence is reflected in status + blockers. |
Comment on lines
+144
to
+149
| const eventCounts = {}; | ||
| const latestByComponentEvent = {}; | ||
|
|
||
| for (const record of records) { | ||
| if (!record.component || !record.event_name) continue; | ||
| eventCounts[record.component] ||= {}; |
Comment on lines
+115
to
+131
| function parseCsvList(value) { | ||
| if (Array.isArray(value)) { | ||
| return value.map(cleanString).filter(Boolean); | ||
| } | ||
| return cleanString(value) | ||
| .split(',') | ||
| .map(cleanString) | ||
| .filter(Boolean); | ||
| } | ||
|
|
||
| function summarizeOrganicEvidence(records = [], options = {}) { | ||
| const requiredEvents = parseCsvList( | ||
| options.required_organic_events ?? | ||
| options.requiredOrganicEvents ?? | ||
| process.env.BOT_COMMENT_AUTH_REQUIRED_ORGANIC_EVENTS | ||
| ); | ||
| const requiredComponents = parseCsvList( |
Comment on lines
+128
to
+137
| - name: Upload campaign state | ||
| if: ${{ always() }} | ||
| uses: actions/upload-artifact@v7 | ||
| with: | ||
| name: sync-dependabot-campaign-state-${{ github.run_id }} | ||
| path: | | ||
| sync-dependabot-campaign-state.json | ||
| sync-dependabot-campaign-summary.md | ||
| if-no-files-found: warn | ||
| retention-days: 30 |
Comment on lines
+176
to
+184
| return { | ||
| schema: 'workflows-bot-comment-auth-organic-evidence/v1', | ||
| required_events: requiredEvents, | ||
| required_components: components, | ||
| expected_mode: expectedMode === 'unknown' ? '' : expectedMode, | ||
| event_counts: eventCounts, | ||
| blockers, | ||
| status: blockers.length > 0 ? 'warning' : 'pass', | ||
| }; |
stranske
temporarily deployed
to
agent-high-privilege
April 25, 2026 17:46 — with
GitHub Actions
Inactive
stranske
force-pushed
the
codex/workflow-telemetry-contracts
branch
from
April 25, 2026 17:53
eb611e2 to
d579872
Compare
stranske
temporarily deployed
to
agent-high-privilege
April 25, 2026 17:53 — with
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated Status Summary
Scope
Sync/Dependabot Campaign Queue
Remote discovery found more review-thread work than fits in a full GitHub issue body. The marker below retains the compact machine-readable queue for the local watcher.
Context for Agent
Related Issues/PRs
Tasks
Acceptance criteria
Head SHA: d579872
Latest Runs: ✅ success — Gate
Required: gate: ✅ success