fix(image-refresh): a bookkeeping write must not abort the tick (backend#2007)

[LukasWodka]

Summary

Closes the second half of the image-refresh-cronjob.yaml pair. backend#2007.

The settled branch clears tracebloc.io/refresh-skip-streak under set -eu with no guard (client/templates/image-refresh-cronjob.yaml:360). The read two lines above is deliberately non-fatal; the write was left fatal — and the comment shows the asymmetry was never decided: it reasons about the read and says nothing about the write.

That matters because the clear runs before Pass 1. A transient API error on one bookkeeping annotation aborts the tick, so requests-proxy and resource-monitor stay frozen on the very tick that proved jobs-manager had recovered.

	what happens	operator sees
backend#1964 (already on prod)	a never-Ready jobs-manager makes the tick a benign skip, exit 0	green CronJob, no image updates
this	once it settles, the annotation clear can abort the tick before Pass 1	red Job, no image updates

So #1964's own recovery path had its own way to stay frozen.

The change

Make the clear non-fatal and report it. Nothing below reads SKIP_KEY, so a stale value costs exactly one thing: the next unsettled tick resumes counting from the old number instead of zero, tripping the ceiling earlier — fail-safe in the direction #1964 wants.

Logged as a WARNING rather than swallowed, because the shared lesson of both findings is that "images did not update" must never be inferable only from the Job's colour.

Scoped to this one call site on purpose. I checked the other five kubectl annotate calls; all are deliberately fatal with recorded reasoning — the streak record fails closed so the ceiling cannot be dodged (#1964), the attempt counter must persist before a restart, and the success reset is split from the digest write precisely so a failure there cannot stick a flap lockout (#626). Only this one is pure cleanup.

Tests

Extends #1964's harness, which renders the chart, extracts the script the pod actually runs, and executes it against a stubbed kubectl — STUB_ANNOTATE_RC was already a knob.

The assertion is that Pass 1 is reached (its per-image "checking <repo>" lines), not the exit status: a status assertion would also pass if the script died later for an unrelated reason.

Mutation-proved. With the template hunk reverted, both new cases fail:

not ok 1 a failed streak clear does NOT abort the tick before Pass 1 (backend#2007)
# `grep -q 'checking tracebloc/jobs-manager' <<<"$output" || return 1' failed
not ok 2 a failed streak clear is reported as a WARNING, not swallowed (backend#2007)
# `grep -q 'WARNING: could not clear tracebloc.io/refresh-skip-streak' ...' failed

Case 1 failing at checking tracebloc/jobs-manager is the bug reproduced — the tick really did abort before Pass 1. The mutation anchor was confirmed absent from the reverted file rather than assumed.

16/16 green with the fix restored; helm lint clean; Chart.yaml bumped 1.9.43 → 1.9.44 (chart-version-guard run locally against the real base).

Why this is on the critical path

This is the one genuine fix among the three findings holding the client leg of the 2026-08-14 staging hop. The other two need no code: backend#2008 is refuted on the arithmetic (Helm's daemonSetReady tolerates Desired - maxUnavailable, and maxUnavailable defaults to 1), and the AMD thread is refuted against the upstream manifest.

Related: backend#2007, backend#1964, backend#2008, client#719, client#718.

---

Note

Low Risk
Single guarded annotate in the image-refresh script with documented fail-safe stale-annotation behavior; other annotate sites stay fatal by design.

Overview
When jobs-manager settles after a skip streak, clearing tracebloc.io/refresh-skip-streak no longer aborts the CronJob tick under set -e. A failed kubectl annotate is logged as WARNING and execution continues into Pass 1, so requests-proxy and resource-monitor can refresh on the recovery tick instead of failing red with no image updates.

Stale skip-streak values only shorten the next streak ceiling (fail-safe vs. #1964). Chart version 1.9.44; bats cover “reaches Pass 1 after failed clear” and “WARNING is visible.”

^{Reviewed by Cursor Bugbot for commit 7f9c9dd. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 7f9c9dd. Configure here.}