fix(cronjobs): a failing tick must leave a Pod somebody can read (backend#2620)

[LukasWodka]

Closes tracebloc/backend#2620

The defect

Both CronJobs the chart renders used restartPolicy: OnFailure. Kubernetes deletes an OnFailure Job's Pod once backoffLimit is exhausted, and failedJobsHistoryLimit retains the Job — it cannot retain a Pod that no longer exists.

So a repeatedly-failing CronJob leaves a row of Failed Jobs and no logs at all. The upstream docs say it plainly:

"your Pod running the Job will be terminated once the job backoff limit has been reached. This can make debugging the Job's executable more difficult. We suggest setting restartPolicy = 'Never'"

Measured, not theoretical

An edge's auto-upgrade CronJob had been failing hourly for 2.7 days. Five Failed Jobs were retained; every one of their Pods was gone, and the namespace event window had rolled past the first failure — so the reason was unrecoverable from the cluster.

The only surviving evidence that anything was wrong was the CronJob's lastSuccessfulTime. Each Job failed 39 seconds in, so this was never a timeout or a hang: the logs would have said exactly what it was, and they were thrown away.

The consequence was that the edge silently stopped accepting chart upgrades — including any security fix — while looking healthy from every angle anyone checks.

Both CronJobs, not just the one that broke

image-refresh carried the same policy on a 15-minute schedule — the same blind spot at four times the frequency. Fixing only the instance that happened to fail would have left the class.

The guard

scripts/tests/cronjob-failures-are-readable.sh reads every CronJob out of the rendered manifests, so a third one is covered the day it lands rather than when someone remembers. It holds no list of CronJob names.

Fails closed twice:

• zero CronJobs rendered is a refusal, not a clean sweep — a guard that checks nothing otherwise passes;
• a template declaring kind: CronJob that no value combination in the render matrix reaches is reported UNREACHED by name and fails, rather than passing silently as "nothing to check".

Wired into DRIFT_GUARDS (now 21 entries — the drift target counts its iterations and refuses to report green on fewer, so adding a guard without wiring it cannot pass).

Scoped to CronJob deliberately, with the argument recorded at the declaration rather than assumed: a Helm hook Job loses its Pod the same way, but its failure fails the release and is reported to whoever ran it — somebody is already looking. A CronJob's failure is reported to nobody and repeats forever, which is what makes unreadable logs fatal rather than inconvenient. Widening it would need that argument made.

Mutation evidence

Four mutations, each run with the baseline confirmed green afterwards:

mutation	result
one template reverts to OnFailure	FAIL — names the template and the policy
restartPolicy removed entirely	FAIL — "silence is the defect" (Kubernetes defaults it to OnFailure)
render matrix produces no CronJobs	FAIL — refuses to report on zero
a new CronJob template goes unreached	FAIL — names it as [UNREACHED]

On the two tests that changed

auto_upgrade_test.yaml and image_refresh_test.yaml each explicitly asserted OnFailure. They were right to pin it — a policy that matters should be pinned — so they are updated rather than deleted, and the comment now records why the pinned value changed and where the class-level sweep lives.

Checks

• make helm-unittest — 579 passed, 34 suites
• make drift — all 21 guards green (verified the new one actually executed, not just that the count matched)
• make helm-lint, make helm-template — clean
• shellcheck on the new guard — clean

Note on rollout

This change only takes effect on an edge once that edge receives the chart. An edge whose auto-upgrade is already wedged cannot pull it — so unwedging those is separate operational work, tracked in the ticket. The immediate mitigation there is a direct kubectl patch of the CronJob's restartPolicy, which this change makes permanent and unnecessary going forward.

Ticket: tracebloc/backend#2620

---

Note

Low Risk
Operational observability fix for scheduled jobs with no auth or data-path changes; slightly more failed Pods retained per Job tick, which is intentional.

Overview
Fixes backend#2620: chart CronJobs used restartPolicy: OnFailure, which lets Kubernetes delete the Pod once backoffLimit is exhausted while failedJobsHistoryLimit only keeps the Job—so repeated failures left no logs (observed on a prod edge with hourly auto-upgrade failures for 2.7 days).

auto-upgrade and image-refresh CronJob pod templates now set restartPolicy: Never so each attempt keeps a Pod for debugging. Chart bumps to 1.9.74.

A new required drift guard scripts/tests/cronjob-failures-are-readable.sh renders client/ingestor charts and requires every rendered CronJob to declare Never (or fails on missing policy / unreached CronJob templates). It is wired into DRIFT_GUARDS. Helm unittest suites for both CronJobs pin Never instead of OnFailure.

Rollout note: edges already wedged on auto-upgrade still need operational unwedging; this only applies after they receive the chart (or a kubectl patch on the CronJob).

^{Reviewed by Cursor Bugbot for commit 92b0f6e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saadqbal]

The measurement is what makes this a bug report rather than a style note: five Failed Jobs retained, every Pod gone, the event window rolled past the first failure, and each Job failing 39 seconds in — so not a hang, and the logs would have said exactly what it was. "The reason was unrecoverable from the cluster" is the sentence that matters, and the consequence is worse than the mechanism: an edge silently stopped accepting chart upgrades, security fixes included, while looking healthy from every angle anyone checks.

Fixing image-refresh too — same policy, four times the frequency — is the right instinct, and "fixing only the instance that happened to fail would have left the class" is exactly the failure I flagged on client-runtime#417 an hour ago, where a guard existed in two sibling publishers and was missing from the one shipping the control plane. Good to see it applied ahead of the incident rather than after.

The guard fails closed in three places, not the two you claimed. I checked each:

• if not seen: sys.exit("the render matrix produced ZERO CronJobs. Nothing was checked, and a guard that checks nothing passes — refusing.")
• unreached = expected - seen, each reported by name with the reason its policy went unchecked, and folded into the exit condition
• and the one the description doesn't mention: a Pod spec stating norestartPolicy at all lands in missing and fails, so an unstated policy isn't read as an acceptable one

The declared set coming from grep -rlE '^kind: CronJob$' is line-anchored, so a template that merely mentions CronJob in a comment can't inflate expected and manufacture a phantom UNREACHED. And the [OK] line printing both len(seen) and len(expected) means a passing run is falsifiable rather than just green.

Two judgement calls I'd have made the same way. Scoping to CronJob and recording the argument at the declaration — a Helm hook Job loses its Pod identically, but its failure fails the release in front of the person who ran it, whereas a CronJob's failure is reported to nobody and repeats forever. And the failure message telling a future maintainer to "make that argument at the declaration and narrow this guard deliberately rather than adding an exception list here" — which is the right lesson to encode, given .github just spent three PRs burning an exemption list down to empty.

Green, no threads. 👍

[LukasWodka]

Closes tracebloc/backend#2620

The defect

Both CronJobs the chart renders used restartPolicy: OnFailure. Kubernetes deletes an OnFailure Job's Pod once backoffLimit is exhausted, and failedJobsHistoryLimit retains the Job — it cannot retain a Pod that no longer exists.

So a repeatedly-failing CronJob leaves a row of Failed Jobs and no logs at all. The upstream docs say it plainly:

"your Pod running the Job will be terminated once the job backoff limit has been reached. This can make debugging the Job's executable more difficult. We suggest setting restartPolicy = 'Never'"

Measured, not theoretical

An edge's auto-upgrade CronJob had been failing hourly for 2.7 days. Five Failed Jobs were retained; every one of their Pods was gone, and the namespace event window had rolled past the first failure — so the reason was unrecoverable from the cluster.

The only surviving evidence that anything was wrong was the CronJob's lastSuccessfulTime. Each Job failed 39 seconds in, so this was never a timeout or a hang: the logs would have said exactly what it was, and they were thrown away.

The consequence was that the edge silently stopped accepting chart upgrades — including any security fix — while looking healthy from every angle anyone checks.

Both CronJobs, not just the one that broke

image-refresh carried the same policy on a 15-minute schedule — the same blind spot at four times the frequency. Fixing only the instance that happened to fail would have left the class.

The guard

scripts/tests/cronjob-failures-are-readable.sh reads every CronJob out of the rendered manifests, so a third one is covered the day it lands rather than when someone remembers. It holds no list of CronJob names.

Fails closed twice:

• zero CronJobs rendered is a refusal, not a clean sweep — a guard that checks nothing otherwise passes;
• a template declaring kind: CronJob that no value combination in the render matrix reaches is reported UNREACHED by name and fails, rather than passing silently as "nothing to check".

Wired into DRIFT_GUARDS (now 21 entries — the drift target counts its iterations and refuses to report green on fewer, so adding a guard without wiring it cannot pass).

Scoped to CronJob deliberately, with the argument recorded at the declaration rather than assumed: a Helm hook Job loses its Pod the same way, but its failure fails the release and is reported to whoever ran it — somebody is already looking. A CronJob's failure is reported to nobody and repeats forever, which is what makes unreadable logs fatal rather than inconvenient. Widening it would need that argument made.

Mutation evidence

Four mutations, each run with the baseline confirmed green afterwards:

mutation	result
one template reverts to OnFailure	FAIL — names the template and the policy
restartPolicy removed entirely	FAIL — "silence is the defect" (Kubernetes defaults it to OnFailure)
render matrix produces no CronJobs	FAIL — refuses to report on zero
a new CronJob template goes unreached	FAIL — names it as [UNREACHED]

On the two tests that changed

auto_upgrade_test.yaml and image_refresh_test.yaml each explicitly asserted OnFailure. They were right to pin it — a policy that matters should be pinned — so they are updated rather than deleted, and the comment now records why the pinned value changed and where the class-level sweep lives.

Checks

• make helm-unittest — 579 passed, 34 suites
• make drift — all 21 guards green (verified the new one actually executed, not just that the count matched)
• make helm-lint, make helm-template — clean
• shellcheck on the new guard — clean

Note on rollout

This change only takes effect on an edge once that edge receives the chart. An edge whose auto-upgrade is already wedged cannot pull it — so unwedging those is separate operational work, tracked in the ticket. The immediate mitigation there is a direct kubectl patch of the CronJob's restartPolicy, which this change makes permanent and unnecessary going forward.

Ticket: tracebloc/backend#2620

---

Note

Low Risk
Operational observability fix for scheduled jobs with no auth or data-path changes; slightly more failed Pods retained per Job tick, which is intentional.

Overview
Fixes backend#2620: chart CronJobs used restartPolicy: OnFailure, which lets Kubernetes delete the Pod once backoffLimit is exhausted while failedJobsHistoryLimit only keeps the Job—so repeated failures left no logs (observed on a prod edge with hourly auto-upgrade failures for 2.7 days).

auto-upgrade and image-refresh CronJob pod templates now set restartPolicy: Never so each attempt keeps a Pod for debugging. Chart bumps to 1.9.74.

A new required drift guard scripts/tests/cronjob-failures-are-readable.sh renders client/ingestor charts and requires every rendered CronJob to declare Never (or fails on missing policy / unreached CronJob templates). It is wired into DRIFT_GUARDS. Helm unittest suites for both CronJobs pin Never instead of OnFailure.

Rollout note: edges already wedged on auto-upgrade still need operational unwedging; this only applies after they receive the chart (or a kubectl patch on the CronJob).

^{Reviewed by Cursor Bugbot for commit 92b0f6e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saadqbal]

The measurement is what makes this a bug report rather than a style note: five Failed Jobs retained, every Pod gone, the event window rolled past the first failure, and each Job failing 39 seconds in — so not a hang, and the logs would have said exactly what it was. "The reason was unrecoverable from the cluster" is the sentence that matters, and the consequence is worse than the mechanism: an edge silently stopped accepting chart upgrades, security fixes included, while looking healthy from every angle anyone checks.

Fixing image-refresh too — same policy, four times the frequency — is the right instinct, and "fixing only the instance that happened to fail would have left the class" is exactly the failure I flagged on client-runtime#417 an hour ago, where a guard existed in two sibling publishers and was missing from the one shipping the control plane. Good to see it applied ahead of the incident rather than after.

The guard fails closed in three places, not the two you claimed. I checked each:

• if not seen: sys.exit("the render matrix produced ZERO CronJobs. Nothing was checked, and a guard that checks nothing passes — refusing.")
• unreached = expected - seen, each reported by name with the reason its policy went unchecked, and folded into the exit condition
• and the one the description doesn't mention: a Pod spec stating norestartPolicy at all lands in missing and fails, so an unstated policy isn't read as an acceptable one

The declared set coming from grep -rlE '^kind: CronJob$' is line-anchored, so a template that merely mentions CronJob in a comment can't inflate expected and manufacture a phantom UNREACHED. And the [OK] line printing both len(seen) and len(expected) means a passing run is falsifiable rather than just green.

Two judgement calls I'd have made the same way. Scoping to CronJob and recording the argument at the declaration — a Helm hook Job loses its Pod identically, but its failure fails the release in front of the person who ran it, whereas a CronJob's failure is reported to nobody and repeats forever. And the failure message telling a future maintainer to "make that argument at the declaration and narrow this guard deliberately rather than adding an exception list here" — which is the right lesson to encode, given .github just spent three PRs burning an exemption list down to empty.

Green, no threads. 👍

[LukasWodka]

Closes tracebloc/backend#2620

The defect

Both CronJobs the chart renders used restartPolicy: OnFailure. Kubernetes deletes an OnFailure Job's Pod once backoffLimit is exhausted, and failedJobsHistoryLimit retains the Job — it cannot retain a Pod that no longer exists.

So a repeatedly-failing CronJob leaves a row of Failed Jobs and no logs at all. The upstream docs say it plainly:

"your Pod running the Job will be terminated once the job backoff limit has been reached. This can make debugging the Job's executable more difficult. We suggest setting restartPolicy = 'Never'"

Measured, not theoretical

An edge's auto-upgrade CronJob had been failing hourly for 2.7 days. Five Failed Jobs were retained; every one of their Pods was gone, and the namespace event window had rolled past the first failure — so the reason was unrecoverable from the cluster.

The only surviving evidence that anything was wrong was the CronJob's lastSuccessfulTime. Each Job failed 39 seconds in, so this was never a timeout or a hang: the logs would have said exactly what it was, and they were thrown away.

The consequence was that the edge silently stopped accepting chart upgrades — including any security fix — while looking healthy from every angle anyone checks.

Both CronJobs, not just the one that broke

image-refresh carried the same policy on a 15-minute schedule — the same blind spot at four times the frequency. Fixing only the instance that happened to fail would have left the class.

The guard

scripts/tests/cronjob-failures-are-readable.sh reads every CronJob out of the rendered manifests, so a third one is covered the day it lands rather than when someone remembers. It holds no list of CronJob names.

Fails closed twice:

• zero CronJobs rendered is a refusal, not a clean sweep — a guard that checks nothing otherwise passes;
• a template declaring kind: CronJob that no value combination in the render matrix reaches is reported UNREACHED by name and fails, rather than passing silently as "nothing to check".

Wired into DRIFT_GUARDS (now 21 entries — the drift target counts its iterations and refuses to report green on fewer, so adding a guard without wiring it cannot pass).

Scoped to CronJob deliberately, with the argument recorded at the declaration rather than assumed: a Helm hook Job loses its Pod the same way, but its failure fails the release and is reported to whoever ran it — somebody is already looking. A CronJob's failure is reported to nobody and repeats forever, which is what makes unreadable logs fatal rather than inconvenient. Widening it would need that argument made.

Mutation evidence

Four mutations, each run with the baseline confirmed green afterwards:

mutation	result
one template reverts to OnFailure	FAIL — names the template and the policy
restartPolicy removed entirely	FAIL — "silence is the defect" (Kubernetes defaults it to OnFailure)
render matrix produces no CronJobs	FAIL — refuses to report on zero
a new CronJob template goes unreached	FAIL — names it as [UNREACHED]

On the two tests that changed

auto_upgrade_test.yaml and image_refresh_test.yaml each explicitly asserted OnFailure. They were right to pin it — a policy that matters should be pinned — so they are updated rather than deleted, and the comment now records why the pinned value changed and where the class-level sweep lives.

Checks

• make helm-unittest — 579 passed, 34 suites
• make drift — all 21 guards green (verified the new one actually executed, not just that the count matched)
• make helm-lint, make helm-template — clean
• shellcheck on the new guard — clean

Note on rollout

This change only takes effect on an edge once that edge receives the chart. An edge whose auto-upgrade is already wedged cannot pull it — so unwedging those is separate operational work, tracked in the ticket. The immediate mitigation there is a direct kubectl patch of the CronJob's restartPolicy, which this change makes permanent and unnecessary going forward.

Ticket: tracebloc/backend#2620

---

Note

Low Risk
Operational observability fix for scheduled jobs with no auth or data-path changes; slightly more failed Pods retained per Job tick, which is intentional.

Overview
Fixes backend#2620: chart CronJobs used restartPolicy: OnFailure, which lets Kubernetes delete the Pod once backoffLimit is exhausted while failedJobsHistoryLimit only keeps the Job—so repeated failures left no logs (observed on a prod edge with hourly auto-upgrade failures for 2.7 days).

auto-upgrade and image-refresh CronJob pod templates now set restartPolicy: Never so each attempt keeps a Pod for debugging. Chart bumps to 1.9.74.

A new required drift guard scripts/tests/cronjob-failures-are-readable.sh renders client/ingestor charts and requires every rendered CronJob to declare Never (or fails on missing policy / unreached CronJob templates). It is wired into DRIFT_GUARDS. Helm unittest suites for both CronJobs pin Never instead of OnFailure.

Rollout note: edges already wedged on auto-upgrade still need operational unwedging; this only applies after they receive the chart (or a kubectl patch on the CronJob).

^{Reviewed by Cursor Bugbot for commit 92b0f6e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saadqbal]

The measurement is what makes this a bug report rather than a style note: five Failed Jobs retained, every Pod gone, the event window rolled past the first failure, and each Job failing 39 seconds in — so not a hang, and the logs would have said exactly what it was. "The reason was unrecoverable from the cluster" is the sentence that matters, and the consequence is worse than the mechanism: an edge silently stopped accepting chart upgrades, security fixes included, while looking healthy from every angle anyone checks.

Fixing image-refresh too — same policy, four times the frequency — is the right instinct, and "fixing only the instance that happened to fail would have left the class" is exactly the failure I flagged on client-runtime#417 an hour ago, where a guard existed in two sibling publishers and was missing from the one shipping the control plane. Good to see it applied ahead of the incident rather than after.

The guard fails closed in three places, not the two you claimed. I checked each:

• if not seen: sys.exit("the render matrix produced ZERO CronJobs. Nothing was checked, and a guard that checks nothing passes — refusing.")
• unreached = expected - seen, each reported by name with the reason its policy went unchecked, and folded into the exit condition
• and the one the description doesn't mention: a Pod spec stating norestartPolicy at all lands in missing and fails, so an unstated policy isn't read as an acceptable one

The declared set coming from grep -rlE '^kind: CronJob$' is line-anchored, so a template that merely mentions CronJob in a comment can't inflate expected and manufacture a phantom UNREACHED. And the [OK] line printing both len(seen) and len(expected) means a passing run is falsifiable rather than just green.

Two judgement calls I'd have made the same way. Scoping to CronJob and recording the argument at the declaration — a Helm hook Job loses its Pod identically, but its failure fails the release in front of the person who ran it, whereas a CronJob's failure is reported to nobody and repeats forever. And the failure message telling a future maintainer to "make that argument at the declaration and narrow this guard deliberately rather than adding an exception list here" — which is the right lesson to encode, given .github just spent three PRs burning an exemption list down to empty.

Green, no threads. 👍

[LukasWodka]

Closes tracebloc/backend#2620

The defect

Both CronJobs the chart renders used restartPolicy: OnFailure. Kubernetes deletes an OnFailure Job's Pod once backoffLimit is exhausted, and failedJobsHistoryLimit retains the Job — it cannot retain a Pod that no longer exists.

So a repeatedly-failing CronJob leaves a row of Failed Jobs and no logs at all. The upstream docs say it plainly:

"your Pod running the Job will be terminated once the job backoff limit has been reached. This can make debugging the Job's executable more difficult. We suggest setting restartPolicy = 'Never'"

Measured, not theoretical

An edge's auto-upgrade CronJob had been failing hourly for 2.7 days. Five Failed Jobs were retained; every one of their Pods was gone, and the namespace event window had rolled past the first failure — so the reason was unrecoverable from the cluster.

The only surviving evidence that anything was wrong was the CronJob's lastSuccessfulTime. Each Job failed 39 seconds in, so this was never a timeout or a hang: the logs would have said exactly what it was, and they were thrown away.

The consequence was that the edge silently stopped accepting chart upgrades — including any security fix — while looking healthy from every angle anyone checks.

Both CronJobs, not just the one that broke

image-refresh carried the same policy on a 15-minute schedule — the same blind spot at four times the frequency. Fixing only the instance that happened to fail would have left the class.

The guard

scripts/tests/cronjob-failures-are-readable.sh reads every CronJob out of the rendered manifests, so a third one is covered the day it lands rather than when someone remembers. It holds no list of CronJob names.

Fails closed twice:

• zero CronJobs rendered is a refusal, not a clean sweep — a guard that checks nothing otherwise passes;
• a template declaring kind: CronJob that no value combination in the render matrix reaches is reported UNREACHED by name and fails, rather than passing silently as "nothing to check".

Wired into DRIFT_GUARDS (now 21 entries — the drift target counts its iterations and refuses to report green on fewer, so adding a guard without wiring it cannot pass).

Scoped to CronJob deliberately, with the argument recorded at the declaration rather than assumed: a Helm hook Job loses its Pod the same way, but its failure fails the release and is reported to whoever ran it — somebody is already looking. A CronJob's failure is reported to nobody and repeats forever, which is what makes unreadable logs fatal rather than inconvenient. Widening it would need that argument made.

Mutation evidence

Four mutations, each run with the baseline confirmed green afterwards:

mutation	result
one template reverts to OnFailure	FAIL — names the template and the policy
restartPolicy removed entirely	FAIL — "silence is the defect" (Kubernetes defaults it to OnFailure)
render matrix produces no CronJobs	FAIL — refuses to report on zero
a new CronJob template goes unreached	FAIL — names it as [UNREACHED]

On the two tests that changed

auto_upgrade_test.yaml and image_refresh_test.yaml each explicitly asserted OnFailure. They were right to pin it — a policy that matters should be pinned — so they are updated rather than deleted, and the comment now records why the pinned value changed and where the class-level sweep lives.

Checks

• make helm-unittest — 579 passed, 34 suites
• make drift — all 21 guards green (verified the new one actually executed, not just that the count matched)
• make helm-lint, make helm-template — clean
• shellcheck on the new guard — clean

Note on rollout

This change only takes effect on an edge once that edge receives the chart. An edge whose auto-upgrade is already wedged cannot pull it — so unwedging those is separate operational work, tracked in the ticket. The immediate mitigation there is a direct kubectl patch of the CronJob's restartPolicy, which this change makes permanent and unnecessary going forward.

Ticket: tracebloc/backend#2620

---

Note

Low Risk
Operational observability fix for scheduled jobs with no auth or data-path changes; slightly more failed Pods retained per Job tick, which is intentional.

Overview
Fixes backend#2620: chart CronJobs used restartPolicy: OnFailure, which lets Kubernetes delete the Pod once backoffLimit is exhausted while failedJobsHistoryLimit only keeps the Job—so repeated failures left no logs (observed on a prod edge with hourly auto-upgrade failures for 2.7 days).

auto-upgrade and image-refresh CronJob pod templates now set restartPolicy: Never so each attempt keeps a Pod for debugging. Chart bumps to 1.9.74.

A new required drift guard scripts/tests/cronjob-failures-are-readable.sh renders client/ingestor charts and requires every rendered CronJob to declare Never (or fails on missing policy / unreached CronJob templates). It is wired into DRIFT_GUARDS. Helm unittest suites for both CronJobs pin Never instead of OnFailure.

Rollout note: edges already wedged on auto-upgrade still need operational unwedging; this only applies after they receive the chart (or a kubectl patch on the CronJob).

^{Reviewed by Cursor Bugbot for commit 92b0f6e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saadqbal]

The measurement is what makes this a bug report rather than a style note: five Failed Jobs retained, every Pod gone, the event window rolled past the first failure, and each Job failing 39 seconds in — so not a hang, and the logs would have said exactly what it was. "The reason was unrecoverable from the cluster" is the sentence that matters, and the consequence is worse than the mechanism: an edge silently stopped accepting chart upgrades, security fixes included, while looking healthy from every angle anyone checks.

Fixing image-refresh too — same policy, four times the frequency — is the right instinct, and "fixing only the instance that happened to fail would have left the class" is exactly the failure I flagged on client-runtime#417 an hour ago, where a guard existed in two sibling publishers and was missing from the one shipping the control plane. Good to see it applied ahead of the incident rather than after.

The guard fails closed in three places, not the two you claimed. I checked each:

• if not seen: sys.exit("the render matrix produced ZERO CronJobs. Nothing was checked, and a guard that checks nothing passes — refusing.")
• unreached = expected - seen, each reported by name with the reason its policy went unchecked, and folded into the exit condition
• and the one the description doesn't mention: a Pod spec stating norestartPolicy at all lands in missing and fails, so an unstated policy isn't read as an acceptable one

The declared set coming from grep -rlE '^kind: CronJob$' is line-anchored, so a template that merely mentions CronJob in a comment can't inflate expected and manufacture a phantom UNREACHED. And the [OK] line printing both len(seen) and len(expected) means a passing run is falsifiable rather than just green.

Two judgement calls I'd have made the same way. Scoping to CronJob and recording the argument at the declaration — a Helm hook Job loses its Pod identically, but its failure fails the release in front of the person who ran it, whereas a CronJob's failure is reported to nobody and repeats forever. And the failure message telling a future maintainer to "make that argument at the declaration and narrow this guard deliberately rather than adding an exception list here" — which is the right lesson to encode, given .github just spent three PRs burning an exemption list down to empty.

Green, no threads. 👍

[LukasWodka]

Closes tracebloc/backend#2620

The defect

Both CronJobs the chart renders used restartPolicy: OnFailure. Kubernetes deletes an OnFailure Job's Pod once backoffLimit is exhausted, and failedJobsHistoryLimit retains the Job — it cannot retain a Pod that no longer exists.

So a repeatedly-failing CronJob leaves a row of Failed Jobs and no logs at all. The upstream docs say it plainly:

"your Pod running the Job will be terminated once the job backoff limit has been reached. This can make debugging the Job's executable more difficult. We suggest setting restartPolicy = 'Never'"

Measured, not theoretical

An edge's auto-upgrade CronJob had been failing hourly for 2.7 days. Five Failed Jobs were retained; every one of their Pods was gone, and the namespace event window had rolled past the first failure — so the reason was unrecoverable from the cluster.

The only surviving evidence that anything was wrong was the CronJob's lastSuccessfulTime. Each Job failed 39 seconds in, so this was never a timeout or a hang: the logs would have said exactly what it was, and they were thrown away.

The consequence was that the edge silently stopped accepting chart upgrades — including any security fix — while looking healthy from every angle anyone checks.

Both CronJobs, not just the one that broke

image-refresh carried the same policy on a 15-minute schedule — the same blind spot at four times the frequency. Fixing only the instance that happened to fail would have left the class.

The guard

scripts/tests/cronjob-failures-are-readable.sh reads every CronJob out of the rendered manifests, so a third one is covered the day it lands rather than when someone remembers. It holds no list of CronJob names.

Fails closed twice:

• zero CronJobs rendered is a refusal, not a clean sweep — a guard that checks nothing otherwise passes;
• a template declaring kind: CronJob that no value combination in the render matrix reaches is reported UNREACHED by name and fails, rather than passing silently as "nothing to check".

Wired into DRIFT_GUARDS (now 21 entries — the drift target counts its iterations and refuses to report green on fewer, so adding a guard without wiring it cannot pass).

Scoped to CronJob deliberately, with the argument recorded at the declaration rather than assumed: a Helm hook Job loses its Pod the same way, but its failure fails the release and is reported to whoever ran it — somebody is already looking. A CronJob's failure is reported to nobody and repeats forever, which is what makes unreadable logs fatal rather than inconvenient. Widening it would need that argument made.

Mutation evidence

Four mutations, each run with the baseline confirmed green afterwards:

mutation	result
one template reverts to OnFailure	FAIL — names the template and the policy
restartPolicy removed entirely	FAIL — "silence is the defect" (Kubernetes defaults it to OnFailure)
render matrix produces no CronJobs	FAIL — refuses to report on zero
a new CronJob template goes unreached	FAIL — names it as [UNREACHED]

On the two tests that changed

auto_upgrade_test.yaml and image_refresh_test.yaml each explicitly asserted OnFailure. They were right to pin it — a policy that matters should be pinned — so they are updated rather than deleted, and the comment now records why the pinned value changed and where the class-level sweep lives.

Checks

• make helm-unittest — 579 passed, 34 suites
• make drift — all 21 guards green (verified the new one actually executed, not just that the count matched)
• make helm-lint, make helm-template — clean
• shellcheck on the new guard — clean

Note on rollout

This change only takes effect on an edge once that edge receives the chart. An edge whose auto-upgrade is already wedged cannot pull it — so unwedging those is separate operational work, tracked in the ticket. The immediate mitigation there is a direct kubectl patch of the CronJob's restartPolicy, which this change makes permanent and unnecessary going forward.

Ticket: tracebloc/backend#2620

---

Note

Low Risk
Operational observability fix for scheduled jobs with no auth or data-path changes; slightly more failed Pods retained per Job tick, which is intentional.

Overview
Fixes backend#2620: chart CronJobs used restartPolicy: OnFailure, which lets Kubernetes delete the Pod once backoffLimit is exhausted while failedJobsHistoryLimit only keeps the Job—so repeated failures left no logs (observed on a prod edge with hourly auto-upgrade failures for 2.7 days).

auto-upgrade and image-refresh CronJob pod templates now set restartPolicy: Never so each attempt keeps a Pod for debugging. Chart bumps to 1.9.74.

A new required drift guard scripts/tests/cronjob-failures-are-readable.sh renders client/ingestor charts and requires every rendered CronJob to declare Never (or fails on missing policy / unreached CronJob templates). It is wired into DRIFT_GUARDS. Helm unittest suites for both CronJobs pin Never instead of OnFailure.

Rollout note: edges already wedged on auto-upgrade still need operational unwedging; this only applies after they receive the chart (or a kubectl patch on the CronJob).

^{Reviewed by Cursor Bugbot for commit 92b0f6e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saadqbal]

The measurement is what makes this a bug report rather than a style note: five Failed Jobs retained, every Pod gone, the event window rolled past the first failure, and each Job failing 39 seconds in — so not a hang, and the logs would have said exactly what it was. "The reason was unrecoverable from the cluster" is the sentence that matters, and the consequence is worse than the mechanism: an edge silently stopped accepting chart upgrades, security fixes included, while looking healthy from every angle anyone checks.

Fixing image-refresh too — same policy, four times the frequency — is the right instinct, and "fixing only the instance that happened to fail would have left the class" is exactly the failure I flagged on client-runtime#417 an hour ago, where a guard existed in two sibling publishers and was missing from the one shipping the control plane. Good to see it applied ahead of the incident rather than after.

The guard fails closed in three places, not the two you claimed. I checked each:

• if not seen: sys.exit("the render matrix produced ZERO CronJobs. Nothing was checked, and a guard that checks nothing passes — refusing.")
• unreached = expected - seen, each reported by name with the reason its policy went unchecked, and folded into the exit condition
• and the one the description doesn't mention: a Pod spec stating norestartPolicy at all lands in missing and fails, so an unstated policy isn't read as an acceptable one

The declared set coming from grep -rlE '^kind: CronJob$' is line-anchored, so a template that merely mentions CronJob in a comment can't inflate expected and manufacture a phantom UNREACHED. And the [OK] line printing both len(seen) and len(expected) means a passing run is falsifiable rather than just green.

Two judgement calls I'd have made the same way. Scoping to CronJob and recording the argument at the declaration — a Helm hook Job loses its Pod identically, but its failure fails the release in front of the person who ran it, whereas a CronJob's failure is reported to nobody and repeats forever. And the failure message telling a future maintainer to "make that argument at the declaration and narrow this guard deliberately rather than adding an exception list here" — which is the right lesson to encode, given .github just spent three PRs burning an exemption list down to empty.

Green, no threads. 👍

[LukasWodka]

Closes tracebloc/backend#2620

The defect

Both CronJobs the chart renders used restartPolicy: OnFailure. Kubernetes deletes an OnFailure Job's Pod once backoffLimit is exhausted, and failedJobsHistoryLimit retains the Job — it cannot retain a Pod that no longer exists.

So a repeatedly-failing CronJob leaves a row of Failed Jobs and no logs at all. The upstream docs say it plainly:

"your Pod running the Job will be terminated once the job backoff limit has been reached. This can make debugging the Job's executable more difficult. We suggest setting restartPolicy = 'Never'"

Measured, not theoretical

An edge's auto-upgrade CronJob had been failing hourly for 2.7 days. Five Failed Jobs were retained; every one of their Pods was gone, and the namespace event window had rolled past the first failure — so the reason was unrecoverable from the cluster.

The only surviving evidence that anything was wrong was the CronJob's lastSuccessfulTime. Each Job failed 39 seconds in, so this was never a timeout or a hang: the logs would have said exactly what it was, and they were thrown away.

The consequence was that the edge silently stopped accepting chart upgrades — including any security fix — while looking healthy from every angle anyone checks.

Both CronJobs, not just the one that broke

image-refresh carried the same policy on a 15-minute schedule — the same blind spot at four times the frequency. Fixing only the instance that happened to fail would have left the class.

The guard

scripts/tests/cronjob-failures-are-readable.sh reads every CronJob out of the rendered manifests, so a third one is covered the day it lands rather than when someone remembers. It holds no list of CronJob names.

Fails closed twice:

• zero CronJobs rendered is a refusal, not a clean sweep — a guard that checks nothing otherwise passes;
• a template declaring kind: CronJob that no value combination in the render matrix reaches is reported UNREACHED by name and fails, rather than passing silently as "nothing to check".

Wired into DRIFT_GUARDS (now 21 entries — the drift target counts its iterations and refuses to report green on fewer, so adding a guard without wiring it cannot pass).

Scoped to CronJob deliberately, with the argument recorded at the declaration rather than assumed: a Helm hook Job loses its Pod the same way, but its failure fails the release and is reported to whoever ran it — somebody is already looking. A CronJob's failure is reported to nobody and repeats forever, which is what makes unreadable logs fatal rather than inconvenient. Widening it would need that argument made.

Mutation evidence

Four mutations, each run with the baseline confirmed green afterwards:

mutation	result
one template reverts to OnFailure	FAIL — names the template and the policy
restartPolicy removed entirely	FAIL — "silence is the defect" (Kubernetes defaults it to OnFailure)
render matrix produces no CronJobs	FAIL — refuses to report on zero
a new CronJob template goes unreached	FAIL — names it as [UNREACHED]

On the two tests that changed

auto_upgrade_test.yaml and image_refresh_test.yaml each explicitly asserted OnFailure. They were right to pin it — a policy that matters should be pinned — so they are updated rather than deleted, and the comment now records why the pinned value changed and where the class-level sweep lives.

Checks

• make helm-unittest — 579 passed, 34 suites
• make drift — all 21 guards green (verified the new one actually executed, not just that the count matched)
• make helm-lint, make helm-template — clean
• shellcheck on the new guard — clean

Note on rollout

This change only takes effect on an edge once that edge receives the chart. An edge whose auto-upgrade is already wedged cannot pull it — so unwedging those is separate operational work, tracked in the ticket. The immediate mitigation there is a direct kubectl patch of the CronJob's restartPolicy, which this change makes permanent and unnecessary going forward.

Ticket: tracebloc/backend#2620

---

Note

Low Risk
Operational observability fix for scheduled jobs with no auth or data-path changes; slightly more failed Pods retained per Job tick, which is intentional.

Overview
Fixes backend#2620: chart CronJobs used restartPolicy: OnFailure, which lets Kubernetes delete the Pod once backoffLimit is exhausted while failedJobsHistoryLimit only keeps the Job—so repeated failures left no logs (observed on a prod edge with hourly auto-upgrade failures for 2.7 days).

auto-upgrade and image-refresh CronJob pod templates now set restartPolicy: Never so each attempt keeps a Pod for debugging. Chart bumps to 1.9.74.

A new required drift guard scripts/tests/cronjob-failures-are-readable.sh renders client/ingestor charts and requires every rendered CronJob to declare Never (or fails on missing policy / unreached CronJob templates). It is wired into DRIFT_GUARDS. Helm unittest suites for both CronJobs pin Never instead of OnFailure.

Rollout note: edges already wedged on auto-upgrade still need operational unwedging; this only applies after they receive the chart (or a kubectl patch on the CronJob).

^{Reviewed by Cursor Bugbot for commit 92b0f6e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saadqbal]

The measurement is what makes this a bug report rather than a style note: five Failed Jobs retained, every Pod gone, the event window rolled past the first failure, and each Job failing 39 seconds in — so not a hang, and the logs would have said exactly what it was. "The reason was unrecoverable from the cluster" is the sentence that matters, and the consequence is worse than the mechanism: an edge silently stopped accepting chart upgrades, security fixes included, while looking healthy from every angle anyone checks.

Fixing image-refresh too — same policy, four times the frequency — is the right instinct, and "fixing only the instance that happened to fail would have left the class" is exactly the failure I flagged on client-runtime#417 an hour ago, where a guard existed in two sibling publishers and was missing from the one shipping the control plane. Good to see it applied ahead of the incident rather than after.

The guard fails closed in three places, not the two you claimed. I checked each:

• if not seen: sys.exit("the render matrix produced ZERO CronJobs. Nothing was checked, and a guard that checks nothing passes — refusing.")
• unreached = expected - seen, each reported by name with the reason its policy went unchecked, and folded into the exit condition
• and the one the description doesn't mention: a Pod spec stating norestartPolicy at all lands in missing and fails, so an unstated policy isn't read as an acceptable one

The declared set coming from grep -rlE '^kind: CronJob$' is line-anchored, so a template that merely mentions CronJob in a comment can't inflate expected and manufacture a phantom UNREACHED. And the [OK] line printing both len(seen) and len(expected) means a passing run is falsifiable rather than just green.

Two judgement calls I'd have made the same way. Scoping to CronJob and recording the argument at the declaration — a Helm hook Job loses its Pod identically, but its failure fails the release in front of the person who ran it, whereas a CronJob's failure is reported to nobody and repeats forever. And the failure message telling a future maintainer to "make that argument at the declaration and narrow this guard deliberately rather than adding an exception list here" — which is the right lesson to encode, given .github just spent three PRs burning an exemption list down to empty.

Green, no threads. 👍

[LukasWodka]

Closes tracebloc/backend#2620

The defect

Both CronJobs the chart renders used restartPolicy: OnFailure. Kubernetes deletes an OnFailure Job's Pod once backoffLimit is exhausted, and failedJobsHistoryLimit retains the Job — it cannot retain a Pod that no longer exists.

So a repeatedly-failing CronJob leaves a row of Failed Jobs and no logs at all. The upstream docs say it plainly:

"your Pod running the Job will be terminated once the job backoff limit has been reached. This can make debugging the Job's executable more difficult. We suggest setting restartPolicy = 'Never'"

Measured, not theoretical

An edge's auto-upgrade CronJob had been failing hourly for 2.7 days. Five Failed Jobs were retained; every one of their Pods was gone, and the namespace event window had rolled past the first failure — so the reason was unrecoverable from the cluster.

The only surviving evidence that anything was wrong was the CronJob's lastSuccessfulTime. Each Job failed 39 seconds in, so this was never a timeout or a hang: the logs would have said exactly what it was, and they were thrown away.

The consequence was that the edge silently stopped accepting chart upgrades — including any security fix — while looking healthy from every angle anyone checks.

Both CronJobs, not just the one that broke

image-refresh carried the same policy on a 15-minute schedule — the same blind spot at four times the frequency. Fixing only the instance that happened to fail would have left the class.

The guard

scripts/tests/cronjob-failures-are-readable.sh reads every CronJob out of the rendered manifests, so a third one is covered the day it lands rather than when someone remembers. It holds no list of CronJob names.

Fails closed twice:

• zero CronJobs rendered is a refusal, not a clean sweep — a guard that checks nothing otherwise passes;
• a template declaring kind: CronJob that no value combination in the render matrix reaches is reported UNREACHED by name and fails, rather than passing silently as "nothing to check".

Wired into DRIFT_GUARDS (now 21 entries — the drift target counts its iterations and refuses to report green on fewer, so adding a guard without wiring it cannot pass).

Scoped to CronJob deliberately, with the argument recorded at the declaration rather than assumed: a Helm hook Job loses its Pod the same way, but its failure fails the release and is reported to whoever ran it — somebody is already looking. A CronJob's failure is reported to nobody and repeats forever, which is what makes unreadable logs fatal rather than inconvenient. Widening it would need that argument made.

Mutation evidence

Four mutations, each run with the baseline confirmed green afterwards:

mutation	result
one template reverts to OnFailure	FAIL — names the template and the policy
restartPolicy removed entirely	FAIL — "silence is the defect" (Kubernetes defaults it to OnFailure)
render matrix produces no CronJobs	FAIL — refuses to report on zero
a new CronJob template goes unreached	FAIL — names it as [UNREACHED]

On the two tests that changed

auto_upgrade_test.yaml and image_refresh_test.yaml each explicitly asserted OnFailure. They were right to pin it — a policy that matters should be pinned — so they are updated rather than deleted, and the comment now records why the pinned value changed and where the class-level sweep lives.

Checks

• make helm-unittest — 579 passed, 34 suites
• make drift — all 21 guards green (verified the new one actually executed, not just that the count matched)
• make helm-lint, make helm-template — clean
• shellcheck on the new guard — clean

Note on rollout

This change only takes effect on an edge once that edge receives the chart. An edge whose auto-upgrade is already wedged cannot pull it — so unwedging those is separate operational work, tracked in the ticket. The immediate mitigation there is a direct kubectl patch of the CronJob's restartPolicy, which this change makes permanent and unnecessary going forward.

Ticket: tracebloc/backend#2620

---

Note

Low Risk
Operational observability fix for scheduled jobs with no auth or data-path changes; slightly more failed Pods retained per Job tick, which is intentional.

Overview
Fixes backend#2620: chart CronJobs used restartPolicy: OnFailure, which lets Kubernetes delete the Pod once backoffLimit is exhausted while failedJobsHistoryLimit only keeps the Job—so repeated failures left no logs (observed on a prod edge with hourly auto-upgrade failures for 2.7 days).

auto-upgrade and image-refresh CronJob pod templates now set restartPolicy: Never so each attempt keeps a Pod for debugging. Chart bumps to 1.9.74.

A new required drift guard scripts/tests/cronjob-failures-are-readable.sh renders client/ingestor charts and requires every rendered CronJob to declare Never (or fails on missing policy / unreached CronJob templates). It is wired into DRIFT_GUARDS. Helm unittest suites for both CronJobs pin Never instead of OnFailure.

Rollout note: edges already wedged on auto-upgrade still need operational unwedging; this only applies after they receive the chart (or a kubectl patch on the CronJob).

^{Reviewed by Cursor Bugbot for commit 92b0f6e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saadqbal]

The measurement is what makes this a bug report rather than a style note: five Failed Jobs retained, every Pod gone, the event window rolled past the first failure, and each Job failing 39 seconds in — so not a hang, and the logs would have said exactly what it was. "The reason was unrecoverable from the cluster" is the sentence that matters, and the consequence is worse than the mechanism: an edge silently stopped accepting chart upgrades, security fixes included, while looking healthy from every angle anyone checks.

Fixing image-refresh too — same policy, four times the frequency — is the right instinct, and "fixing only the instance that happened to fail would have left the class" is exactly the failure I flagged on client-runtime#417 an hour ago, where a guard existed in two sibling publishers and was missing from the one shipping the control plane. Good to see it applied ahead of the incident rather than after.

The guard fails closed in three places, not the two you claimed. I checked each:

• if not seen: sys.exit("the render matrix produced ZERO CronJobs. Nothing was checked, and a guard that checks nothing passes — refusing.")
• unreached = expected - seen, each reported by name with the reason its policy went unchecked, and folded into the exit condition
• and the one the description doesn't mention: a Pod spec stating norestartPolicy at all lands in missing and fails, so an unstated policy isn't read as an acceptable one

The declared set coming from grep -rlE '^kind: CronJob$' is line-anchored, so a template that merely mentions CronJob in a comment can't inflate expected and manufacture a phantom UNREACHED. And the [OK] line printing both len(seen) and len(expected) means a passing run is falsifiable rather than just green.

Two judgement calls I'd have made the same way. Scoping to CronJob and recording the argument at the declaration — a Helm hook Job loses its Pod identically, but its failure fails the release in front of the person who ran it, whereas a CronJob's failure is reported to nobody and repeats forever. And the failure message telling a future maintainer to "make that argument at the declaration and narrow this guard deliberately rather than adding an exception list here" — which is the right lesson to encode, given .github just spent three PRs burning an exemption list down to empty.

Green, no threads. 👍

[LukasWodka]

Closes tracebloc/backend#2620

The defect

Both CronJobs the chart renders used restartPolicy: OnFailure. Kubernetes deletes an OnFailure Job's Pod once backoffLimit is exhausted, and failedJobsHistoryLimit retains the Job — it cannot retain a Pod that no longer exists.

So a repeatedly-failing CronJob leaves a row of Failed Jobs and no logs at all. The upstream docs say it plainly:

"your Pod running the Job will be terminated once the job backoff limit has been reached. This can make debugging the Job's executable more difficult. We suggest setting restartPolicy = 'Never'"

Measured, not theoretical

An edge's auto-upgrade CronJob had been failing hourly for 2.7 days. Five Failed Jobs were retained; every one of their Pods was gone, and the namespace event window had rolled past the first failure — so the reason was unrecoverable from the cluster.

The only surviving evidence that anything was wrong was the CronJob's lastSuccessfulTime. Each Job failed 39 seconds in, so this was never a timeout or a hang: the logs would have said exactly what it was, and they were thrown away.

The consequence was that the edge silently stopped accepting chart upgrades — including any security fix — while looking healthy from every angle anyone checks.

Both CronJobs, not just the one that broke

image-refresh carried the same policy on a 15-minute schedule — the same blind spot at four times the frequency. Fixing only the instance that happened to fail would have left the class.

The guard

scripts/tests/cronjob-failures-are-readable.sh reads every CronJob out of the rendered manifests, so a third one is covered the day it lands rather than when someone remembers. It holds no list of CronJob names.

Fails closed twice:

• zero CronJobs rendered is a refusal, not a clean sweep — a guard that checks nothing otherwise passes;
• a template declaring kind: CronJob that no value combination in the render matrix reaches is reported UNREACHED by name and fails, rather than passing silently as "nothing to check".

Wired into DRIFT_GUARDS (now 21 entries — the drift target counts its iterations and refuses to report green on fewer, so adding a guard without wiring it cannot pass).

Scoped to CronJob deliberately, with the argument recorded at the declaration rather than assumed: a Helm hook Job loses its Pod the same way, but its failure fails the release and is reported to whoever ran it — somebody is already looking. A CronJob's failure is reported to nobody and repeats forever, which is what makes unreadable logs fatal rather than inconvenient. Widening it would need that argument made.

Mutation evidence

Four mutations, each run with the baseline confirmed green afterwards:

mutation	result
one template reverts to OnFailure	FAIL — names the template and the policy
restartPolicy removed entirely	FAIL — "silence is the defect" (Kubernetes defaults it to OnFailure)
render matrix produces no CronJobs	FAIL — refuses to report on zero
a new CronJob template goes unreached	FAIL — names it as [UNREACHED]

On the two tests that changed

auto_upgrade_test.yaml and image_refresh_test.yaml each explicitly asserted OnFailure. They were right to pin it — a policy that matters should be pinned — so they are updated rather than deleted, and the comment now records why the pinned value changed and where the class-level sweep lives.

Checks

• make helm-unittest — 579 passed, 34 suites
• make drift — all 21 guards green (verified the new one actually executed, not just that the count matched)
• make helm-lint, make helm-template — clean
• shellcheck on the new guard — clean

Note on rollout

This change only takes effect on an edge once that edge receives the chart. An edge whose auto-upgrade is already wedged cannot pull it — so unwedging those is separate operational work, tracked in the ticket. The immediate mitigation there is a direct kubectl patch of the CronJob's restartPolicy, which this change makes permanent and unnecessary going forward.

Ticket: tracebloc/backend#2620

---

Note

Low Risk
Operational observability fix for scheduled jobs with no auth or data-path changes; slightly more failed Pods retained per Job tick, which is intentional.

Overview
Fixes backend#2620: chart CronJobs used restartPolicy: OnFailure, which lets Kubernetes delete the Pod once backoffLimit is exhausted while failedJobsHistoryLimit only keeps the Job—so repeated failures left no logs (observed on a prod edge with hourly auto-upgrade failures for 2.7 days).

auto-upgrade and image-refresh CronJob pod templates now set restartPolicy: Never so each attempt keeps a Pod for debugging. Chart bumps to 1.9.74.

A new required drift guard scripts/tests/cronjob-failures-are-readable.sh renders client/ingestor charts and requires every rendered CronJob to declare Never (or fails on missing policy / unreached CronJob templates). It is wired into DRIFT_GUARDS. Helm unittest suites for both CronJobs pin Never instead of OnFailure.

Rollout note: edges already wedged on auto-upgrade still need operational unwedging; this only applies after they receive the chart (or a kubectl patch on the CronJob).

^{Reviewed by Cursor Bugbot for commit 92b0f6e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saadqbal]

The measurement is what makes this a bug report rather than a style note: five Failed Jobs retained, every Pod gone, the event window rolled past the first failure, and each Job failing 39 seconds in — so not a hang, and the logs would have said exactly what it was. "The reason was unrecoverable from the cluster" is the sentence that matters, and the consequence is worse than the mechanism: an edge silently stopped accepting chart upgrades, security fixes included, while looking healthy from every angle anyone checks.

Fixing image-refresh too — same policy, four times the frequency — is the right instinct, and "fixing only the instance that happened to fail would have left the class" is exactly the failure I flagged on client-runtime#417 an hour ago, where a guard existed in two sibling publishers and was missing from the one shipping the control plane. Good to see it applied ahead of the incident rather than after.

The guard fails closed in three places, not the two you claimed. I checked each:

• if not seen: sys.exit("the render matrix produced ZERO CronJobs. Nothing was checked, and a guard that checks nothing passes — refusing.")
• unreached = expected - seen, each reported by name with the reason its policy went unchecked, and folded into the exit condition
• and the one the description doesn't mention: a Pod spec stating norestartPolicy at all lands in missing and fails, so an unstated policy isn't read as an acceptable one

The declared set coming from grep -rlE '^kind: CronJob$' is line-anchored, so a template that merely mentions CronJob in a comment can't inflate expected and manufacture a phantom UNREACHED. And the [OK] line printing both len(seen) and len(expected) means a passing run is falsifiable rather than just green.

Two judgement calls I'd have made the same way. Scoping to CronJob and recording the argument at the declaration — a Helm hook Job loses its Pod identically, but its failure fails the release in front of the person who ran it, whereas a CronJob's failure is reported to nobody and repeats forever. And the failure message telling a future maintainer to "make that argument at the declaration and narrow this guard deliberately rather than adding an exception list here" — which is the right lesson to encode, given .github just spent three PRs burning an exemption list down to empty.

Green, no threads. 👍