[reliability] Daily Reliability Review - 2026-08-04

[github-actions]

Executive Summary

Over the last 24 hours, the gh-aw project logged 22 failed workflow runs across 13 distinct workflows (via gh-aw.run.status:failure on the spans dataset). No cancellations were detected, and no explicit timeout markers were found. Two workflows account for half of all failures: Contribution Check (6 failures) and PR Code Quality Reviewer (5 failures). Separately, this review surfaced a real instrumentation gap: gen_ai.response.finish_reasons and the release/service.version resource attribute were never populated on any span in the 24h window, even though the emitter code intends to always attach finish_reasons to the run-completion span. The Sentry errors and logs datasets were both completely empty for the window, meaning the 22 span-level failures are not surfacing as Sentry Issues.

Top Reliability Findings

Priority	Workflow	Problem	Evidence	Next Action
High	Contribution Check	6 distinct failed runs in 24h — highest volume	6 unique trace IDs with gh-aw.run.status:failure (spans dataset, 24h)	Investigate shared root cause (recent config/prompt change, rate limit, MCP tool failure)
High	PR Code Quality Reviewer	5 distinct failed runs in 24h	5 unique trace IDs with gh-aw.run.status:failure; representative trace 1c7f1791c24245ec6b191bca735fa12b shows failure spans interleaved with success spans in the same trace	Check for a flaky dependency causing partial retries within a single run
Medium	[aw] Failure Investigator (6h)	The reliability-monitoring workflow itself failed (1 run, 6 failure-status spans)	Trace 6cef8e028802dbcc4f4c8a498f847c86	Prioritize — a broken failure-investigator can mask other incidents
Medium	10 other workflows (Linter Miner, Daily Issues Report Generator, Daily Agent of the Day Blog Writer, CI Optimization Coach, Daily File Diet, Claude Code User Documentation Review, Typist - Go Type Analysis, Glossary Maintainer, Daily Skill Optimizer Improvements, Daily Model Inventory Checker)	1 failed run each	Span-level gh-aw.run.status:failure, 24h window	Monitor; single occurrences, not yet a recurring pattern
Instrumentation gap	All workflows	gen_ai.response.finish_reasons never populated despite code intending to always emit it on the run-completion span	has:gen_ai.response.finish_reasons → 0 results across 24h	Cannot distinguish timeout vs. other failure causes; audit OTLP export/indexing for this array attribute
Instrumentation gap	All workflows	Release correlation attribute (service.version/release) never populated	has:release, has:sentry.release → 0 results (24h)	Cannot correlate failures to a specific gh-aw release; verify resource-attribute mapping end to end
Observability gap	N/A	errors dataset empty (0 events) while 22 span-level failures exist	dataset=errors, 24h → 0 results	Failed runs aren't creating Sentry Issues — consider explicit error capture at the OTLP exporter for alerting/triage
Observability gap	N/A	logs dataset empty (0 events)	dataset=logs, 24h → 0 results	No structured log ingestion active for this project in this window

Representative Traces

View representative traces

• Contribution Check (failure): trace 940682b7ae8e238664e271ff11fcdd4b
• PR Code Quality Reviewer (failure, mixed with success spans in-trace): trace 1c7f1791c24245ec6b191bca735fa12b
• [aw] Failure Investigator (6h) (failure): trace 6cef8e028802dbcc4f4c8a498f847c86
• Typist - Go Type Analysis (failure, single run with 6 failure-status spans): trace 7975a67c361fde2f3cdc3c7ee92d3592
• Claude Code User Documentation Review (failure, single run with 6 failure-status spans): trace a52f6d7fb1fd458dbba0871eace2a759

Note: several traces above emit multiple gh-aw.run.status:failure spans per single logical run (a "hop" per span rather than per run), which is why raw span counts were deduplicated by trace ID before ranking workflows.

Recommendations

• Investigate Contribution Check and PR Code Quality Reviewer first — together they account for 11 of 22 failures (50%) in the window; look for a shared cause (recent prompt/config change, MCP tool outage, or rate limiting).
• Fix the [aw] Failure Investigator (6h) workflow — the tool meant to catch reliability issues is itself failing, which risks masking other incidents.
• Close the gen_ai.response.finish_reasons gap — the emitter (actions/setup/js/send_otlp_span.cjs) is designed to always attach this attribute to the run-completion span, but it was absent on every span sampled in 24h. Audit whether the attribute is being dropped in export, or not indexed as a searchable field on the Sentry side.
• Wire failed runs into the errors dataset — 22 confirmed span-level failures produced zero Sentry error events, so today's triage relied entirely on span queries rather than issue-based alerting.
• Restore release correlation — service.version/release was unpopulated on every span, making it impossible to tie failures to a specific gh-aw release; verify the resource-attribute mapping end to end (emit side already sets service.version, per the code).

Notes

View notes

• The task brief suggested querying gh_aw.workflow_name (underscore), but that attribute does not exist. The emitter (actions/setup/js/send_otlp_span.cjs) actually emits gh-aw.workflow.name and gh-aw.run.status (dot-separated), which are present and reliably populated — this was cross-checked in source before drawing conclusions and is likely a naming-convention drift between docs and code.
• span.status is present and reliably populated (ok/error values observed).
• gen_ai.response.finish_reasons and release/service.version are confirmed absent across all sampled spans in the 24h window — reported as a confirmed instrumentation gap, not a data-availability fluke, since multiple query shapes (has:, explicit field request) all returned empty.
• errors and logs datasets were both empty for the full 24h window — stated explicitly per review protocol rather than silently skipped.
• No span.status:cancelled spans and no spans with timeout/otlp keywords in span.description were found — this is inconclusive on whether any of the 22 failures were specifically timeouts, since the attribute that would disambiguate (gen_ai.response.finish_reasons) is the one confirmed missing.
• Tooling notes: this MCP build does not expose search_events or get_trace_details — list_events was used throughout, with trace:<id> filters as the fallback for trace-continuity checks. Aggregate count() queries consistently returned HTTP 400 (orderby must also be in the selected columns); per the fallback protocol, raw event rows were fetched and deduplicated/aggregated client-side instead of retrying the same query shape.

References:

• Contribution Check failure trace
• PR Code Quality Reviewer failure trace
• Failure Investigator (6h) failure trace

Generated by 🚨 Daily Reliability Review · agent · 121.2 AIC · ⌖ 39.2 AIC · ⊞ 5.9K · ◷

• expires on Aug 6, 2026, 3:25 PM UTC-08:00