[Outcome Report] Outcome Health Report — 2026-06-10

[github-actions]

Workflow Health — 2026-06-10

Executive read: 28 outcomes evaluated across 8 runs; 19 items pending (67%), indicating active processing. Most volume is in Issue Monster and Smoke Claude workflows. Acceptance rate is strong at 71%, but zero-touch rate of 0% indicates all accepted items required human engagement.

Workflow	Status	Lifecycle health	References
Smoke Claude	🟨🟩⬜🟥🟨🟨🟥🟨🟨🟨🟨🟨	🔴 stuck	🟨 38238 · 🟩 comment · ⬜ discussion · 🟥 38197 · 🟨 review
Issue Monster	🟨🟨🟨🟨🟨🟨	🔴 stuck	🟨 38119 · 🟨 38044 · 🟨 38010
Smoke Codex	🟩🟨🟨	🔴 stuck	🟩 comment · 🟨 38239 · 🟨 comment
AI Moderator	🟨	🔴 stuck	—
Changeset Generator	🟨🟩	🟡 in flight	🟨 38197 · 🟩 38197
Design Decision Gate 🏗️	🟨🟩	🟡 in flight	🟨 38197 · 🟩 comment
Daily Sentrux Report	⬜	⚪ underdefined	⬜ 38243
Agent Container Smoke Test	🟩	🟢 resolving	🟩 comment

Legend:

• Status: 🟩 accepted · 🟥 rejected · 🟨 pending · ⬜ unknown
• Lifecycle health: 🟢 resolving · 🟡 in flight · 🟠 aging · 🔴 stuck · ⚪ underdefined

🔴 Action Items

1. Issue Monster is underdefined: All 6 outcomes are pending with no clear acceptance/rejection signal. Review the workflow's prompt and acceptance criteria. The ~56 minute pending age with no follow-up suggests outputs may not align with actual needs.

2. Smoke Claude needs output quality review: Waste rate of 16.7% (2 rejected items) combined with 8 pending and 1 unknown outcome. Only 8% acceptance rate indicates the workflow's output quality or targeting needs improvement.

3. Smoke Codex pending backlog: 2 of 3 items pending (~53 minutes), with no engagement. Very small sample size (3 total), but monitor for resolution.

4. Design Decision Gate has aging pending item: PR Enforce AI credit resolution order; set built-in defaults to 5000 (daily) and 1000 (per-run) #38197 is pending for 1.2+ hours with no review progress. This PR appears to be a focus point across multiple workflows — consider prioritizing its review.

5. Zero-touch acceptance is 0%: All 5 accepted items required human engagement. This indicates agent output quality could be improved to require less human refinement. Consider reviewing output prompts for clarity and validating that agent outputs match user expectations.

6. Data quality note: 1 item (3.6% of outcomes) evaluated with only existence fallback. Minor signal, but indicates one outcome type lacks dedicated evaluator logic.

Detailed metrics, evidence quality, workflow counts, and trends

Outcome Scorecard — 2026-06-10

Metric	Value	Status
Acceptance rate	71%	🟡 60-80%
Zero-touch rate	0%	🔴 <25%
Waste rate	7%	🟢 <10%
Median time to resolution	—	—
Accepted	5 / 28	—
— strong evidence	0	merged, completed, approved
— medium evidence	5	engaged, retained
— weak evidence	0	existence only
Rejected	2	—
Ignored	0	no observable follow-up
Zero-touch	0 / 5	—
Pending	19	—
Runs checked	8	—

Per-Workflow Breakdown

Workflow	Accepted	Rejected	Pending	Unknown	Acceptance	Waste
Smoke Claude	1	2	8	1	8%	16%
AI Moderator	0	0	1	0	0%	0%
Agent Container Smoke Test	1	0	0	0	100%	0%
Changeset Generator	1	0	1	0	50%	0%
Daily Sentrux Report	0	0	0	1	0%	0%
Design Decision Gate 🏗️	1	0	1	0	50%	0%
Issue Monster	0	0	6	0	0%	0%
Smoke Codex	1	0	2	0	33%	0%

Evidence Quality

⚠️ 1 item was evaluated using only a generic existence check (signal: target_exists_only). This contributes to weak evidence and may overstate acceptance. Dedicated evaluators for add_reviewer, submit_pull_request_review, update_issue, update_pull_request, and other types provide stronger evidence.

📊 Measured by Outcome Collector · 60.2 AIC · ⌖ 4.62 AIC

• expires on Jun 16, 2026, 5:05 PM UTC-08:00

[github-actions]

Workflow Health — 2026-06-10

Executive read: 6 pending items across 4 workflows need attention; 3 are aging >30min. Smoke CI resolving well (2 accepted). Two workflows (Copilot PR Conversation NLP Analysis, GitHub API Consumption Report Agent) producing only unknown-status items suggest weak evaluator signal or unclear acceptance criteria. Issue Monster showing healthy pending volume with recent activity.

Workflow	Status	Lifecycle health	References
Issue Monster	🟨🟨🟨	🟡 in flight	🟨 #38298 · 🟨 #38289 · 🟨 #38282
AI Moderator	🟨	🟠 aging	—
Daily A/B Testing Advisor	🟨	🟠 aging	🟨 #38348
Daily Documentation Updater	🟨	🟠 aging	🟨 #38351
Copilot PR Conversation NLP Analysis	⬜	⚪ underdefined	⬜ #38347
GitHub API Consumption Report Agent	⬜	⚪ underdefined	⬜ #38352
Smoke CI	🟩🟩	🟢 resolving	🟩 #38353 · 🟩 #38353

Legend:

• Status: 🟩 accepted · 🟥 rejected · 🟨 pending · ⬜ unknown
• Lifecycle health: 🟢 resolving · 🟡 in flight · 🟠 aging · 🔴 stuck · ⚪ underdefined
• References: one linked item per status emoji, in the same order as the Status column

🔴 Action Items

1. Aging pending items — 3 workflows have items pending >30 minutes (AI Moderator: 31m, Daily A/B Testing Advisor: 27m, Daily Documentation Updater: 18m). These require human review or an explicit timeout/escalation strategy. If these are expected to remain open indefinitely, update acceptance criteria to classify them as "long-lived" rather than "pending".

2. Underdefined workflows — Copilot PR Conversation NLP Analysis and GitHub API Consumption Report Agent produce only unknown-status outcomes (2 items, 20% of total). These workflows likely lack clear acceptance/rejection criteria or the evaluator has insufficient signal. Either refine acceptance rules or assign dedicated human reviewers for validation.

3. Zero-touch rate is 0% — All 2 accepted items required actor follow-up (0/2 zero-touch). While Smoke CI's small sample makes this less actionable, workflows consistently needing human edits suggest agent output quality or targeting could improve.

4. Data quality concern — 2 items (20% of total outcomes) evaluated using only existence checks (signal: target_exists_only). These contribute to unknown/weak outcomes and may understate true acceptance. Assign dedicated evaluators for create_discussion items in GitHub API Consumption Report Agent and Copilot PR Conversation NLP Analysis.

5. Issue Monster pending trend — Issue Monster has 3 items pending with no follow-up engagement (65+ seconds old). Monitor whether these move to accepted/rejected within 24 hours; if not, escalate or refine criteria.

Detailed metrics, evidence quality, workflow counts, and trends

Outcome Scorecard — 2026-06-10

Metric	Value	Status
Acceptance rate	100%	🟢 >80%
Zero-touch rate	0%	🔴 <25%
Waste rate	0%	🟢 <10%
Median time to resolution	3m	—
Accepted	2 / 10	—
— strong evidence	1	merged, completed, approved
— medium evidence	1	engaged, retained
— weak evidence	0	existence only
Rejected	0	—
Ignored	0	no observable follow-up
Unknown	2	weak signal (existence only)
Pending	6	—
Zero-touch	0 / 2	—
Runs checked	7	—

Per-Workflow Breakdown

Workflow	Accepted	Rejected	Ignored	Pending	Unknown	Acceptance	Zero-touch
Smoke CI	2	0	0	0	0	100%	0%
Issue Monster	0	0	0	3	0	—	—
Daily Documentation Updater	0	0	0	1	0	—	—
Daily A/B Testing Advisor	0	0	0	1	0	—	—
AI Moderator	0	0	0	1	0	—	—
Copilot PR Conversation NLP Analysis	0	0	0	0	1	—	—
GitHub API Consumption Report Agent	0	0	0	0	1	—	—

Evidence Quality

⚠️ 2 item(s) were evaluated using only a generic existence check (signal: target_exists_only). These contribute to unknown outcomes and may understate observable value. The workflows producing these items (Copilot PR Conversation NLP Analysis, GitHub API Consumption Report Agent) need stronger evaluation signals — either dedicated reviewers or clearer acceptance rules for create_discussion outputs.

Trend Signal

Comparing to 2026-06-07 baseline: ⬆️ Improving on acceptance rate (100% now vs 50% on 2026-06-07), but ⬇️ increase in pending volume (6 pending now vs 0 on 2026-06-07). This indicates new workflows are running but outputs are not yet resolving. The two workflows showing underdefined behavior (Copilot PR Conversation NLP Analysis, GitHub API Consumption Report Agent) did not appear in the previous cycle, confirming they are newly active and need evaluation framework definition.

📊 Measured by Outcome Collector · 35.1 AIC · ⌖ 4.55 AIC

• expires on Jun 17, 2026, 4:22 AM UTC-08:00

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #38992

View newer issue

---

This action was performed automatically by the Outcome Collector workflow.