Uh oh!
There was an error while loading. Please reload this page.
fix(chart): jobs-manager Recreate + image-refresh no-restart-storm (RWO-PVC prod incident) - #549
Merged
Merged
Conversation
…C deadlock)
Production incident: a client's hi-jobs-manager sat Pending for 4+ hours, blocking dataset
ingestion. Root cause is entirely ours:
jobs-manager mounts two ReadWriteOnce PVCs (client-pvc, client-logs-pvc; local-path,
WaitForFirstConsumer) but used strategy RollingUpdate with maxSurge:1. maxSurge brings up
a SECOND pod before the old one is gone. On a healthy single node this happens to work
(RWO permits two pods on the same node), so it ran for days. After a routine WSL2/Docker
restart the PVCs must re-provision, and now the rollout presents TWO unscheduled consumers
to local-path — WaitForFirstConsumer can't pick one, so the scheduler's VolumeBinding
PreBind times out ("context deadline exceeded") and the pod never schedules. The
image-refresh CronJob then compounds it: it writes its "last-refreshed digest" annotation
only after `kubectl rollout status` SUCCEEDS, which now never happens, so every 15 min it
sees recorded != latest and restarts again — a self-perpetuating loop (19 revisions).
mysql-client starves on the same wedged provisioner. jobs-manager never goes Ready, so its
Service has no endpoints and the ingestor submit job's curl to jobs-manager:8080 fails.
Fix: strategy Recreate (drop maxSurge) — exactly one RWO consumer at a time, so
provisioning and restarts always converge, even after a node restart. This matches
mysql-deployment, which already uses Recreate for the identical RWO-PVC reason.
Regression guard: jobs_manager_test.yaml now asserts strategy.type == Recreate and no
rollingUpdate block, so this can't silently regress.
Chart 1.9.10 -> 1.9.11 (chart content change must bump the version to reach installs).
Follow-ups filed for the compounding factors: image-refresh restart loop, K8S version pin
reaching clients, and node-restart survival.saadqbal
previously approved these changes
Aug 3, 2026
…d (no restart storm) Second half of the RWO-deadlock incident (#546, folded in to avoid a chart-version collision with the Recreate fix — same incident, same 1.9.11 bump). image-refresh records its digest annotation only after `kubectl rollout status` succeeds. When a rollout can't complete (the RWO deadlock, or any stall), the annotation never lands, so every 15-min tick sees recorded != latest and restarts AGAIN — a sequential restart storm (the incident showed 19 restarts). concurrencyPolicy: Forbid only stops OVERLAPPING jobs, not this. Fix: before touching the deployment, `kubectl rollout status --timeout=10s`; if it isn't settled (rollout in progress, or pod stuck Pending on volume binding), log and exit 0 — a restart can't help an unschedulable pod and only churns ReplicaSets, which on a single-node local-path cluster wedges provisioning further. Retry on the next tick once settled. Regression guard in image_refresh_test.yaml asserts the skip is present in the rendered script. (image_refresh_test's 2 pre-existing failures are local helm v4 schema strictness — identical on clean develop; CI's pinned helm v3.15.4 passes the suite.) Closes#546
shujaatTracebloc
commented
Aug 3, 2026
ContributorAuthor
Folded in the #546 fix (image-refresh skip-when-unsettled) — same incident, and folding avoids a chart-version collision (both need the 1.9.10→1.9.11 bump). This PR now Closes #545 and #546:
Both have helm-unittest regression guards. Only the primary incident (#545) + #546 remain in scope here; #547 (k3s drift warning) and #548 (node-local durability) stay separate — see below. |
saadqbal
approved these changes
Aug 3, 2026
LukasWodka
commented
Aug 3, 2026
Contributor
/fr-pass |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix a production RWO-PVC deadlock: jobs-manager →
RecreateIncident: a client's
hi-jobs-managersatPendingfor 4+ hours, blocking dataset ingestion. Root cause is entirely ours — the client's command and YAML were correct.Root cause
jobs-managermounts two ReadWriteOnce PVCs (client-pvc,client-logs-pvc;local-path,WaitForFirstConsumer) but usedstrategy: RollingUpdatewithmaxSurge: 1— a second pod is created before the old one goes away.WaitForFirstConsumercan't pick one → schedulerVolumeBindingPreBind times out (context deadline exceeded) → never schedules.mysqlalready usesRecreatefor the identical RWO-PVC reason;jobs-managerwas an oversight.image-refreshcompounds it: it records its digest annotation only afterrollout statussucceeds — which now never happens — so it re-restarts every 15 min forever (19 revisions), starvingmysqltoo. jobs-manager never Ready → Service has no endpoints → ingestorcurl jobs-manager:8080fails → ingestion post-hook fails.Fix
jobs-manager→strategy: Recreate(dropmaxSurge): exactly one RWO consumer at a time, so provisioning + restarts always converge, even after a node restart.jobs_manager_test.yamlnow assertsstrategy.type == Recreateand norollingUpdateblock.Validation
helm unittest ./client -f tests/jobs_manager_test.yaml→ 28/28 pass (incl. the new Recreate guard);helm templaterenderstype: Recreate.values.schema.jsonthan CI's pinned v3.15.4 — confirmed identical on clean develop, not from this change.)Follow-ups (compounding factors, filed separately)
Closes#545
Note
Medium Risk
Changes rollout behavior for the core jobs-manager Deployment and image-refresh on every install; low blast radius for multi-node clusters but high operational impact where RWO + image-refresh loops were failing ingestion.
Overview
Fixes a production RWO-PVC deadlock by changing jobs-manager from
RollingUpdate(withmaxSurge: 1) tostrategy: Recreate, so only one pod mounts the ReadWriteOnceclient-pvc/client-logs-pvcat a time—matching mysql and avoiding two consumers wedgingWaitForFirstConsumervolume binding on single-node local-path clusters.The image-refresh script now skips a tick when
kubectl rollout statusdoes not succeed within 10s (rollout in progress or pod not Ready), preventing repeatedrollout restartstorms while the deployment is stuck.Chart version 1.9.10 → 1.9.11. Helm unittest regression guards lock Recreate on jobs-manager and the “not settled … skipping” path in image-refresh.
Reviewed by Cursor Bugbot for commit b8ffc53. Bugbot is set up for automated code reviews on this repo. Configure here.