Skip to content

fix(ci): fail closed when the staleness dedup search errors (backend#2702) - #884

Merged
aptracebloc merged 1 commit into
developfrom
fix/2702-staleness-dedup-fail-closed
Aug 27, 2026
Merged

fix(ci): fail closed when the staleness dedup search errors (backend#2702)#884
aptracebloc merged 1 commit into
developfrom
fix/2702-staleness-dedup-fail-closed

Conversation

@aptracebloc

@aptraceblocaptracebloc commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Closes tracebloc/backend#2702

What

scripts/alert-workflow-staleness.sh deduplicates by searching backend for an open issue already carrying the finding's marker, and files a new work-type:bug only when none is found. The search hid its own failures:

fornumin$(gh search issues ... 2>/dev/null);doif gh issue view "$num" ... 2>/dev/null | grep -qF "$marker";then ...

A gh search / gh issue viewerror produced empty output, indistinguishable from a genuine zero-result — so a transient search failure was read as "not yet tracked" and filed a duplicate. On the daily cron that is one new card per stale workflow per day until search recovers (Bugbot Medium, client#876 thread).

Verified the finding still stands on current develop (#876 shipped the ticket, not a fix). Same class as backend#2631's filer-dedup gap — that one lives in the release-train repo (promote-repo.sh / file-hop-findings.py), a separate codebase with no shared helper, so nothing to fix centrally here; gh search issues has a single call site in this repo.

Fix — fail closed

Capture gh's exit status instead of swallowing it. A failed search — or a failed candidate read — now aborts loud (exit 2, temp body cleaned up) rather than proceeding to create. A genuine zero-result (gh exits 0, empty stdout) still green-lights the create, so healthy dedup is unchanged. exit 2 matches the script's existing create-failure path and turns the (un-continue-on-error) watcher step red — the documented "fails loud" contract.

I chose abort over retry: a gh error at this point is systemic (auth / rate-limit / outage), the daily run re-attempts tomorrow, and a red watcher is the intended signal.

Tests

Adds an offline gh stub to scripts/tests/alert-workflow-staleness.bats exercising the real (non-dry-run) path:

  • dedup search error → exit ≠ 0, noissue create
  • dedup candidate-read error → exit ≠ 0, noissue create
  • existing open issue with the marker → skips, no create
  • genuine empty result → still files (a completed check green-lights create)

Evidence

$ bats scripts/tests/alert-workflow-staleness.bats # 9/9 ok (4 new)
$ shellcheck scripts/alert-workflow-staleness.sh # clean
$ bats scripts/tests/check-style.bats # ok
$ bats scripts/tests/bats-hygiene.bats # ok

Header comment and docs/WORKFLOW-STALENESS.md exit-code contract updated to match.

🤖 Generated with Claude Code


Note

Low Risk
CI alerting script and tests only; behavior is stricter on API errors (no duplicate issues) without changing successful dedup or filing paths.

Overview
Fixes duplicate tracking issues when gh search issues or gh issue view fails during dedup: those errors used to look like “no open issue,” so the daily watcher could file a new card every day until search recovered (backend#2702).

alert-workflow-staleness.sh now checks exit status on dedup search and candidate body reads, aborts with exit 2 (and skips create), and documents the fail-closed rule in the script header and docs/WORKFLOW-STALENESS.md. A successful search with zero hits still allows filing.

Tests add a configurable gh stub and four bats cases covering search failure, view failure, happy-path dedup skip, and empty-search still creates.

Reviewed by Cursor Bugbot for commit 9f24da2. Bugbot is set up for automated code reviews on this repo. Configure here.

…2702)
alert-workflow-staleness.sh hid `gh search issues` / `gh issue view`
failures with `2>/dev/null` and read the resulting empty output as
"not yet tracked", so a transient search error filed a fresh duplicate
`work-type:bug` per stale workflow on every daily cron run until search
recovered. Same class as backend#2631's filer-dedup gap (a separate repo,
no shared helper).
Capture gh's exit status instead of swallowing it: a failed search — or a
failed candidate read — now aborts loud (exit 2, temp body cleaned up)
rather than proceeding to create. A genuine zero-result (gh exits 0, empty
stdout) still green-lights the create, so healthy dedup is unchanged.
Adds an offline `gh` stub to the bats suite covering both fail-closed paths
(search error, candidate-read error), the dedup-hit skip, and the
genuine-empty files-anyway case. Updates the script header and
docs/WORKFLOW-STALENESS.md exit-code contract to match.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@saqlainsyed007saqlainsyed007 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 9f24da26. The fix does exactly what it claims and fails closed on the right path.

alert-workflow-staleness.sh dedups before filing a work-type:bug issue by searching backend for the finding's hidden marker. The old code swallowed gh failures (gh search … 2>/dev/null, and gh issue view … 2>/dev/null | grep), so a transient API error produced empty output indistinguishable from a genuine zero-result — read as "not yet tracked" and filing a fresh duplicate every daily run until search recovered (backend#2702). The fix captures gh's exit status via if ! var="$(…)" and aborts exit 2 (cleaning up the temp body) on either a search or a candidate-read error, while a genuine empty result (gh exit 0) still green-lights the create. Splitting gh issue view from the grep (the old pipe masked gh's status under pipefail) is necessary and done correctly.

I checked the fail-closed tests are mutation-proof: against the reverted code both flip to exit 0 with gh issue create reached, so they redden if the guard is deleted — not vacuous — and the happy-path + dedup-skip tests pin the create/skip boundary. It's the sole dedup call site in the repo. CI green (bats 9/9, shellcheck, Bugbot), mergeable, no open threads. LGTM.

One optional, non-blocking follow-up (pre-existing, out of this diff): the dedup-skip branch around :147continues without rm -f "$bodyfile", unlike the other loop-exit paths, so it leaks one mktemp file per already-tracked workflow per run. Worth a tidy sometime; not a reason to hold this.

@aptracebloc
aptracebloc merged commit 38e79c4 into developAug 27, 2026
42 checks passed
@aptracebloc
aptracebloc deleted the fix/2702-staleness-dedup-fail-closed branch August 27, 2026 14:57
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@aptracebloc@saqlainsyed007@LukasWodka