Uh oh!
There was an error while loading. Please reload this page.
fix(ci): watch scheduled workflows for silent staleness; retire dormant windows-e2e nightly (backend#2627) - #868
Conversation
…nt windows-e2e nightly (backend#2627) `Windows e2e (self-hosted)` ran nightly for 22 days and every run was `cancelled` (GitHub's 24h queue-timeout — no `self-hosted, windows, nested-virt` runner is registered). `cancelled` is not a red check and fires no alert, so a job that had NEVER once succeeded read as "the Windows path is covered nightly" for three weeks. Runner state can't be confirmed without org admin (`gh api .../actions/runners` -> 403); the 22-day queue-timeout streak is conclusive that no matching runner is online. Two parts: 1. Durable fix — a generic staleness watch (item 3, the class fix): - scripts/check-workflow-staleness.sh: read-only, test-seamed detector. For every SCHEDULED workflow in the repo it finds the most recent successful scheduled run and flags any whose newest completed scheduled run is non-success and whose last success is >= N days old (default 7). Gating on "newest completed run is non-success" makes a healthy-but-infrequent (e.g. weekly) job cadence-immune. Exits 0 on a finding on purpose — the alert is the filed issue, not a red check (the whole point of #2627 is that a red/cancelled scheduled check is the signal that gets ignored); it exits non-zero only when the watcher itself is broken, which stays loud. - scripts/alert-workflow-staleness.sh: files ONE deduplicated issue per stale workflow into the private catch-all `backend` (CI-health is internal work), labelled work-type:bug so it routes to Ready. Dedup by a hidden fingerprint marker, same shape as the e2e-agent. - .github/workflows/workflow-staleness-watch.yml: daily, GitHub-hosted (so it always runs). Mints a scoped tracebloc-release-train App token (issues:write on backend) for the cross-repo file — same App/pin as add-to-kanban.yml / envelope-contract-drift.yml. dry-run dispatch input. - bats for both scripts (offline, via the stub/clock seams). Class sweep (in the issue) confirms windows-e2e was the only self-hosted-targeting workflow in the org; the watch is generic so the next scheduled job can't rot silently. On its first real run it will file one issue for digest-drift.yml, which has failed on schedule ~13 days running unnoticed — the watch working as intended, triage separately. 2. Retire the phantom (item 2): remove the nightly `schedule:` trigger from windows-e2e.yaml (manual-dispatch only) with a DORMANT banner, and mark docs/WINDOWS-E2E.md dormant. I can't provision a self-hosted nested-virt runner (no infra/org admin), and the credentialed EC2 Windows journey (backend#2619, now on m7i.xlarge + NestedVirtualization=enabled) is the forward coverage — full deletion is the follow-up once it's green. Because it's no longer scheduled, the new watch correctly leaves it out of scope. Verified: shellcheck clean; 17 bats green; both workflows parse; the detector run live against tracebloc/client caught windows-e2e (22d) and digest-drift (13d) while healthy infrequent workflows stayed ok; end-to-end dry-run filed nothing. `make drift` fails only on a pre-existing local helm v4 kubeVersion mismatch (reproduces on clean origin/develop; CI pins v3.15.4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
…(backend#2627) Two review findings on client#868: 1. Bugbot (HIGH): `die` inside `runs_for` could not stop the watcher. The function is called from `$(...)` and the script runs `set -uo pipefail` without `-e`, so `exit 2` only killed the subshell; the parent continued, printed `ok`, and exited 0 — a broken watcher (API/jq failure) looking healthy and filing nothing, the exact fail-OPEN this script claims to prevent. Fix: runs_for now `return`s non-zero on failure and the caller checks each subshell's status and `die`s in the PARENT shell (where exit works). Added a regression test: a corrupt runs payload now exits 2, not a silent `[]`/0. 2. bats-hygiene (test 77): every standalone `[ ]`/`[[ ]]` / `! cmd` assertion in both new bats files now ends in `|| return 1`, so a failing assertion fails its test rather than being advisory (scripts/tests/unenforced-assertions.awk). Verified: unenforced-assertions.awk clean over both files; bats-hygiene.bats 18/18; the two suites 18/18 (incl. the new fail-closed test); shellcheck -S warning -x clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
aptracebloc
commented
Aug 27, 2026
bugbot run |
Uh oh!
There was an error while loading. Please reload this page.
…ackend#2627)
Second Bugbot finding on client#868 (HIGH), plus a robustness follow-on:
1. jq --argjson hit the Linux argv limit. classify_one passed the whole
workflow_runs array as a single `--argjson` argument, and runs_for fetched
FULL run objects (nested repository/head_repository/actor/…). Linux caps one
argv string at 128KiB (MAX_ARG_STRLEN); a daily workflow's ~100-run history
exceeds that within weeks, so jq failed and — now that the watcher fails
closed — it would die on ubuntu (never on macOS; the cap is Linux-only, and
the 4-field stubs never reproduced it). Fixed two ways:
- runs_for now PROJECTS each run to the four fields classify_one actually
reads ({status, conclusion, created_at, html_url}) at the source, so the
payload is tiny and the stubs are the real shape.
- classify_one reads the runs array from STDIN instead of `--argjson`, which
removes the argv ceiling entirely (belt-and-suspenders).
New regression test feeds a >128KiB projected payload (200 padded runs) and
asserts it classifies rather than dies.
2. runs_for now distinguishes a 404 (workflow file present but no registered
workflow / no runs yet — a just-added scheduled workflow) from auth/other
errors: 404 -> treat as an empty history and skip (the same "idle / brand-new"
case classify_one already handles), everything else -> still fail closed.
Without this, adding any new scheduled workflow could redden the whole watcher
on its first tick and mask every other finding.
Verified: shellcheck -S warning -x clean; unenforced-assertions.awk clean;
both suites 19/19 (incl. the large-payload test); a live run against
tracebloc/client flags digest-drift, 404-skips the not-yet-registered new
workflow, and exits 0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>aptracebloc
commented
Aug 27, 2026
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 0d9afef. Configure here.
saadqbal
left a comment
There was a problem hiding this comment.
Twenty-two consecutive cancelled runs reading as "the Windows path is covered nightly" is the cleanest example of the failure this org keeps hitting: a job that has never once succeeded, presenting as coverage, because the absence of a green is not a red. Fixing the class rather than provisioning a runner is the right call.
Four design decisions I checked and would defend:
The exit-code contract is the right way round, and it's the opposite of the obvious one.green = the watcher ran, red = the watcher is broken, and a finding exits 0 — because the whole premise of #2627 is that a red scheduled check is exactly the signal people ignore. So the alert is a filed issue and the only thing that reddens is the watcher's own failure, which GitHub emails about. Distinguishing "no findings" / "findings" / "I could not tell" across exits 0 / 0 / 2-3 is the three-state discipline rather than a boolean, and putting the loud state on the one condition a human will actually receive is the part that makes it work.
GitHub-hosted, "so it always runs, unlike the self-hosted job it watches for." A monitor that shares its subject's failure mode cannot report that subject's absence. That's the same rule as a coverage axis needing to be observable by the party running the test, one layer out.
?event=schedule filters server-side, so a human manually dispatching the workflow and succeeding cannot make a dead schedule look healthy. Easy to omit, and its absence would have made the watcher quietly wrong in precisely the case where someone pokes at a failing job.
Gating on "most recent COMPLETED scheduled run is non-success" and the age threshold makes it cadence-immune — a weekly job isn't flagged for being weekly, and the watcher needs no per-workflow knowledge of intended frequency. That's what keeps it generic rather than a list someone has to maintain.
And "could not confirm" is the right way to report item 1. Listing runners needs admin:org and both endpoints 403 for you, so you said so, said what remains unknown (registered-but-offline vs never provisioned), and separately noted the 22-day streak is conclusive that none is online. I got this wrong myself earlier this week — 404s on branch protection looked like "no required checks" until a control showed my token simply cannot read it. Naming the permission boundary instead of reporting the absence as a finding is the discipline.
The deployment note is the best evidence in the PR: on its first real run it will file one issue for digest-drift.yml, red on schedule ~13 days unnoticed. A class fix that immediately surfaces a second live instance of the class is the strongest possible demonstration. And work-type:bug into backend matches the convention exactly — bugs skip Backlog.
Retiring the phantom rather than deleting it, with deletion deferred until backend#2619 is reliably green, is the correct sequencing, and noting that a dispatch-only job claims no cadence — so the watch rightly ignores it — closes the loop.
Green, no threads, 17 bats tests behind offline seams. 👍
Uh oh!
There was an error while loading. Please reload this page.
Closes tracebloc/backend#2627.
Closes the class in backend#2627: a scheduled workflow that quietly stops producing a
success— the signal is either acancelled(no red check, no alert) or an ignored red — and nobody notices for weeks.The problem
Windows e2e (self-hosted)ran nightly for 22 days and every run wascancelled— GitHub's 24h queue-timeout, because noself-hosted, windows, nested-virtrunner is registered to pick it up.cancelledis not a red check and raises no alert, so a job that has never once succeeded read as "the Windows path is covered nightly" for three weeks.Runner state (item 1) — could not confirm. Listing runners needs org admin;
gh api /orgs/tracebloc/actions/runnersand/repos/tracebloc/client/actions/runnersboth return 403 for me (noadmin:org). The 22-day queue-timeout streak is conclusive that no matching runner is online; an admin should still confirm whether one is registered-but-offline vs. never provisioned.What this PR does
1. Durable fix — a generic staleness watch (item 3, the class fix)
scripts/check-workflow-staleness.sh— read-only, test-seamed detector. For every scheduled workflow in the repo it finds the most recent successful scheduled run and flags any whose newest completed scheduled run is non-successand whose last success is ≥ N days old (default 7). Gating on "newest completed run is non-success" makes a healthy-but-infrequent (e.g. weekly) job cadence-immune. It exits0on a finding on purpose — the alert is the filed issue, not a red check (the whole point of #2627 is that a red/cancelled scheduled check is the signal that gets ignored). It exits non-zero only when the watcher itself is broken, which stays loud.scripts/alert-workflow-staleness.sh— files one deduplicated issue per stale workflow into the private catch-allbackend(CI-health is internal work, per CLAUDE.md), labelledwork-type:bugso it routes straight toReady. Dedup by a hidden fingerprint marker, same shape as the e2e-agent..github/workflows/workflow-staleness-watch.yml— daily, GitHub-hosted (so it always runs, unlike the self-hosted job it watches for). Mints a scopedtracebloc-release-trainApp token (issues:writeonbackend) for the cross-repo file — same App and pin asadd-to-kanban.yml/envelope-contract-drift.yml. Has adry-rundispatch input.STALENESS_RUNS_STUB/STALENESS_NOWseams) — 17 tests.docs/WORKFLOW-STALENESS.md— semantics, the exit-code contract, known limitations, and how to port it to other repos.The class sweep in #2627 confirms
windows-e2ewas the only self-hosted-targeting workflow in the org; the watch is generic so the next scheduled job can't rot silently.2. Retire the phantom (item 2)
Removed the nightly
schedule:trigger fromwindows-e2e.yaml(now manual-dispatch only) with a DORMANT banner, and markeddocs/WINDOWS-E2E.mddormant. I can't provision a self-hosted nested-virt runner (no infra/org admin), and the credentialed EC2 Windows journey (backend#2619, now onm7i.xlarge+--cpu-options NestedVirtualization=enabled) is the forward coverage — full deletion is the follow-up once #2619 is reliably green. Because it's no longer scheduled, the new watch correctly leaves it out of scope (a dispatch-only job claims no cadence).backendissue fordigest-drift.yml, which has failed on schedule ~13 days running, unnoticed (a live instance of the exact "ignored red" mode from backend#2386). That's the watch working as intended — triagedigest-driftseparately.tracebloc-release-trainApp to haveissues:writeonbackend. If it doesn't, the first scheduled run's file step 403s and the job goes red (loud, by design) — grant the scope, don't silence the watcher. Use Run workflow → dry-run = true to exercise detection without filing.Verification (evidence)
shellcheckclean on both scripts;make lintgreen (61 parse / 66 shellcheck).check-workflow-staleness.bats+alert-workflow-staleness.bats).windows-e2e.yamlnow exposes onlyworkflow_dispatch.tracebloc/clientcaughtwindows-e2e(22d, cancelled) anddigest-drift(13d, failure) whileenvelope-contract-drift,installer-tests,stale-backlog-callerstayedok. End-to-end dry-run filed nothing.digest-driftremains flagged — the exact post-merge behavior.make driftfails only on a pre-existing local helm v4kubeVersionmismatch (cronjob-failures-are-readable.sh) that reproduces on a cleanorigin/developtree; CI pins helm v3.15.4. None of my files are read by any drift guard.Self-review (
/code-review high)Four findings, all low-severity and either mitigated+documented or by-design fail-closed: search-index dedup lag (documented, self-healing); the gh dedup/create path is only dry-run unit-tested (inherent to gh code, verified live); sub-daily-cron window cap (widened 50→100, documented; no such cron in the org); and a single workflow's runs-API error reddening the whole watcher (deliberate fail-closed — tolerant would make an auth-wide failure silently green).
🤖 Generated with Claude Code
Note
Medium Risk
Changes CI alerting and cross-repo issue filing (App
issues:writeonbackend); misconfiguration would 403 loudly. Low product/runtime risk — no app code paths.Overview
Addresses backend#2627: scheduled jobs that stop going green without a loud signal (queue-timeout
cancelledruns or ignored reds).Retires the phantom Windows e2e nightly.
windows-e2e.yamldrops itsschedule:cron and is manual-dispatch only, with docs marking the self-hosted job DORMANT (22 days of nightlycancelledwith no nested-virt runner). Real Windows install coverage is pointed at the EC2 journey (backend#2619).Adds a generic staleness watch so the next rotting scheduled workflow cannot hide:
check-workflow-staleness.shscans workflows withschedule:+cron:, flags those whose newest completed scheduled run is non-success and last success is ≥ 7 days (cadence-safe rule).alert-workflow-staleness.shfiles one deduplicatedwork-type:bugissue per stale workflow in privatetracebloc/backend(not another red check).workflow-staleness-watch.ymlonubuntu-latestruns detection, mints a scoped release-train App token for cross-repoissues:write, supports dry-run.docs/WORKFLOW-STALENESS.mddocuments semantics and exit-code contract (detector exits 0 on findings; fails loud only when the watcher itself breaks).Reviewer note: first real run will likely file for
digest-drift.yml(~13d of scheduled failures) — intentional.Reviewed by Cursor Bugbot for commit 0d9afef. Bugbot is set up for automated code reviews on this repo. Configure here.