fix(release): retry the cosign call, bounded and transient-only (backend#2379)

[LukasWodka]

Summary

v0.10.10-rc.2 existed as a git tag with no GitHub Release and no assets for 6h04m because one of release.yml's eight legs hit a sigstore TUF CDN reset. This retries that one network call.

The ordering, which is the whole defect: the tag is the workflow's trigger, so it exists before any signing happens. The Release object is created by publish, which is needs: release — gated on all eight legs. So the tag necessarily precedes the assets, and any single-leg failure leaves a tag with nothing attached. That ordering cannot be reordered away; the only lever is making the flaky call not fail.

Measured on run 32624602531:

event	time
tag v0.10.10-rc.2 pushed (staging hop, PR #554)	2026-08-23 07:04:32Z
seven legs green	07:05:44–07:05:51Z
darwin/arm64 FAILURE — tuf: failed to download 10.root.json … read: connection reset by peer	07:05Z
Aggregate + create GitHub Release SKIPPED	—
gh run rerun --failed → leg green	13:08:32Z
Release finally published, 27 assets	13:08:45Z

6h04m with a tag and no artifacts. Caught by fr-assist's cli smoke battery, not by the build.

What this does not do

The aggregate gate is untouched.publish still requires all eight legs. A release missing darwin/arm64 is worse than no release, and weakening the gate would ship a partially-signed release — strictly worse than the symptom. And the retry is on the cosign call, not the job: a job-level rerun redoes seven good legs and widens the very window this closes.

The retry design

scripts/cosign-retry.sh wraps cosign, forwarding its arguments verbatim.

BOUNDED — at most 3 attempts, backoff 5s then 15s, per-attempt wall-clock cap 120s. Worst case 3×120 + 5 + 15 = 6m20s, inside the job's timeout-minutes: 20, so the retry can never itself be why a leg is killed. The cap is enforced in bash rather than with coreutils timeout, which is absent on macOS — an untestable branch is how a cap stops being real.

CLASSIFIED — only output matching a known transient sigstore/network signature is retried. A rejected OIDC token (401 Unauthorized from Fulcio) or a missing blob fails on the first attempt, with NOT retrying in the log. This is the point the ticket is sharpest about: an unbounded or indiscriminate retry converts a genuine signing refusal into a slow, noisy failure, which is worse than the bug being fixed.

FAIL CLOSED — cosign exiting 0 is a claim; the artifact is the evidence. Every --output-* file named on the command line must exist and be non-empty afterwards, or the leg fails. The file list is derived from the actual argv, not from a second copy of the naming convention. This case is not retried: a zero exit with no signature is not a network symptom.

Verification

scripts/tests/cosign-retry-verify.sh (15 cases, wired into build.yml's Installer (shell) job — hermetic, no network, no tag required) drives the real script with cosign replaced by a PATH shim. Every case asserts the exit status and the number of cosign invocations, so a case cannot pass by failing for a different reason than its name claims. Its failure texts are written independently of the script's pattern list — including the verbatim 2026-08-23 message — because a list checked against itself is blind.

Mutation-proved, 7 mutations. For each: the anchor was asserted to occur exactly once and the mutant asserted to parse before trusting the red — an inert mutation and real coverage are indistinguishable in a log.

mutation	tests that reddened
classifier removed (all failures look transient)	both "genuine refusal is NOT retried" cases
fail-closed artifact check disabled	both "exit 0 with no/empty artifact" cases
default attempt budget 3 → 1	the shipped-defaults case (+ its backoff assertion)
budget comparison -ge → -gt	both exact-attempt-count cases
per-attempt cap disarmed	the hung-attempt case
backoff schedule → constant 0	both delay cases
short-backoff config guard disabled	the config-refusal case

Baseline 15/15, restored 15/15, zero mutation markers left behind.

Also run: bash -n + shellcheck -S warning -x on both scripts (both now in build.yml's shellcheck list), yaml.safe_load + actionlint on both workflows, scripts/file-budget.sh green.

Docs

RELEASE_CHECKLIST.md gains the recovery for "tag exists, no Release at all", preserving release.yml's reasoning that a re-run — or a dispatch at the tag ref — is the only valid rebuild: cosign embeds the run's ref in the keyless identity and the installers trust only @refs/tags/v.*, so a branch dispatch publishes signatures every customer install rejects (Bugbot on promotion #428).

Audit: is this the first instance, or the first noticed?

Ran git tag vs the actual published artifact across the five version-cutting repos named in release-train's PUBLISH-PATHS.md. cli, client and data-ingestors are 1:1 and clean. design-system and tracebloc-py-package create no GitHub Releases by design, so they were checked against their real registries instead.

One other instance found, a different repo and mechanism — reported separately, not fixed here: tracebloc-py-packagev0.18.0 (tagged 2026-08-06) never published.Publish to TestPyPI succeeded, then Verify install from TestPyPI — a sleep 60 followed by a single un-retried pip install — lost the index-propagation race, so publish-pypi was skipped and 0.18.0 exists on neither TestPyPI nor PyPI 18 days later. Same shape as this bug: one un-retried third-party call between the tag and the artifact.

Closes tracebloc/backend#2379

---

Note

Medium Risk
Touches keyless signing in the release path, but keeps the all-legs publish gate and only retries classified transients; a classifier miss could still flake or delay a genuine refusal.

Overview
Stops a single flaky Sigstore sign-blob from leaving a git tag with no GitHub Release. publish still requires all eight matrix legs; the retry is on that one network call, not a job re-run.

scripts/cosign-retry.sh wraps cosign with at most 3 attempts (5s/15s backoff, 120s per-attempt cap), retries only known TUF/CDN/transport failures, and fails closed if a zero exit produces empty --output-* artifacts. Genuine refusals (OIDC 401, missing blob) fail on the first attempt.

A hermetic PATH-shim harness (cosign-retry-verify.sh) is wired into the Installer job, and RELEASE_CHECKLIST.md documents tag-with-no-assets recovery (rerun failed jobs or dispatch at the tag ref).

^{Reviewed by Cursor Bugbot for commit 31f018b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saqlainsyed007]

Approving. Reviewed the cosign-retry wrapper at high effort last pass; CI is now green with no open threads.

scripts/cosign-retry.sh is correct on all three properties it claims: BOUNDED (≤3 attempts, 5s/15s backoff, per-attempt 120s cap enforced in bash via background+poll+TERM/KILL returning 124 — portable to macOS where coreutils timeout is absent; worst case 6m20s < the job's 20m); CLASSIFIED (rc=124 and matched sigstore/network signatures retry, everything else is a genuine refusal and exits on the first attempt); FAIL-CLOSED (every --output-* file, derived from argv, must exist and be non-empty or exit 3 — not retried, since a zero exit with no signature isn't a network symptom). set -uo pipefail without -e is right for a loop that inspects failures, and the backoff-schedule length is guarded. The release-workflow wiring is a minimal cosign → wrapper swap; the all-eight-legs publish gate is untouched. The hermetic 15-case harness asserts both exit status and cosign invocation count with independently-written failure texts — mutation-proof.

Green, mergeable, no threads.