fix(#1144): pin jobs-manager memory requests==limits so a mass restart can't OOM-kill it

[divyasinghds]

Closes tracebloc/backend#1144.

Problem

On a cluster-recovery mass restart, jobs-manager gets OOM-killed (exit 137). Despite the ticket's "CPUKilled" label, exit 137 is a memory SIGKILL — CPU limits only throttle, they never kill. The pod was Burstable (memory requests < limits), so once its working set exceeded the request it was a prime node-OOM victim.

Fix

Pin requests.memory == limits.memory for both jobs-manager containers (api 1Gi, pods-monitor 512Mi). This does not make the pod Guaranteed QoS — it stays Burstable because requests.cpu != limits.cpu. What it does is lower each container's oom_score_adj, so when the node runs out of memory the kernel OOM-killer treats jobs-manager as a less-preferred victim. Same mechanism the chart's mysql already relies on (mysql likewise pins memory and omits its cpu limit). Also raised CPU requests (api 100m→250m, pods-monitor 50m→100m) to avoid startup throttling. Memory limits are unchanged (1Gi / 512Mi) — this only raises the floor, not the ceiling.

Files

• client/templates/jobs-manager-deployment.yaml, client/values.yaml
• client/tests/jobs_manager_test.yaml — resource assertions updated to the pinned values
• client/Chart.yaml — version + appVersion → 1.9.27

Validation

helm lint clean; helm unittest passes (jobs_manager suite + full suite, 354 tests).

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes default resource reservations for a core control-plane deployment, increasing per-node memory scheduling footprint (~768Mi more requested across both containers) but only adjusts Kubernetes QoS/OOM preference, not application logic.

Overview
Addresses cluster-recovery mass restarts where the jobs-manager api and pods-monitor containers were OOM-killed (exit 137) because memory requests sat below limits, making them favored node-OOM victims.

The chart now raises memory requests to match existing limits (api 1Gi, pods-monitor 512Mi) and increases CPU requests (api 100m→250m, pods-monitor 50m→100m) so startup is less throttled under contention. Memory limits are unchanged; pods remain Burstable (CPU request ≠ limit), matching the mysql pinning pattern. Defaults and template comments live in values.yaml and jobs-manager-deployment.yaml; helm unittest expectations were updated. Chart version bumps to 1.9.27.

^{Reviewed by Cursor Bugbot for commit d26d2a9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[aptracebloc]

@divyasinghds — effective OOM fix, and the resource choices are right. One substantive correction before this is merge-ready: the "Guaranteed QoS" claim is inaccurate — the pod stays Burstable.

🟡 The pod is Burstable, not Guaranteed

The PR body, commit message, values.yaml comment, and both template comments all say this yields Guaranteed QoS. Kubernetes grants Guaranteed only when every container has request == limit for both cpu and memory. Here:

container	cpu req / limit	mem req / limit
api	250m / 1000m ✗	1Gi / 1Gi ✓
pods-monitor	100m / 500m ✗	512Mi / 512Mi ✓

cpu request ≠ cpu limit on both, so kubectl describe pod will report QoS Class: Burstable — contradicting the comments.

The fix still works, just not via that mechanism. Raising requests.memory to the limit lowers the container's Burstable oom_score_adj (1000 − 1000·memRequest/nodeAllocatable), and a container whose request equals its limit is never "over request," so the node OOM-killer stops preferring it. That's genuine protection — and it's exactly what mysql does: mysql has no cpu limit at all, so it isn't Guaranteed either. So the "same as mysql" comparison is accurate in mechanism; only the "Guaranteed QoS" label is wrong.

Caveat worth knowing: a truly Guaranteed pod gets oom_score_adj = −997, far below the best Burstable score (floor 2). Since the whole tracebloc pod set on a k3d node is Burstable, the relative re-ordering still saves jobs-manager in practice — but it isn't the maximal protection "Guaranteed" implies.

Suggested fix (wording only, no resource change): reword the four spots to e.g. "memory request pinned to limit → lowest Burstable OOM score, no longer the preferred victim (same requests==limits-on-memory approach mysql uses)." I'd not chase real Guaranteed by pinning cpu request==limit — that would either re-introduce the startup throttle you just removed (cap at 250m/100m) or reserve 1000m/500m (scheduling pressure on small nodes), and mysql doesn't bother. The resource shape as-is is the right call.

✅ Verified correct

• --reuse-values safe: template | default fallbacks updated to match values.yaml (api 250m/1Gi, pods-monitor 100m/512Mi), so an upgrade from a release predating these keys still gets the new floors.
• Only the floor moved — memory limits unchanged (1Gi/512Mi), so headroom can't shrink.
• Security untouched — jobs-manager is control-plane (not a training pod); securityContext unchanged.

⚪ Non-blocking notes

• Scheduling footprint grows to 1.5Gi mem + 350m cpu reserved (was 768Mi + 150m). On a tight single-node k3d that could push toward Pending — worth a sanity check on min node sizing (documented tradeoff).
• Chart 1.9.25 → 1.9.27 skips 1.9.26 (taken by #641). Fine — the guard checks monotonic increase, not contiguity. Heads-up that #637 (→1.9.23) is now behind both and will need a re-bump to ≥1.9.28.

Net: the fix is sound and I'd approve once the "Guaranteed QoS" phrasing is corrected — it's the one thing that will actively mislead the next reader.

🤖 Generated with Claude Code

[divyasinghds]

Thanks @aptracebloc and @saadqbal for the Burstable correction — you're right, the pod is not Guaranteed QoS (cpu request != limit); only memory is pinned. Reworded the framing across the templates, values.yaml, the test description and the PR body: the real protection is the lowered oom_score_adj from requests.memory == limits.memory, which makes jobs-manager a less-preferred node-OOM victim (same pattern mysql uses, which also omits its cpu limit). Resource values are unchanged. Also fixed the failing jobs_manager_test resource assertions to the pinned values (api 1Gi, pods-monitor 512Mi); helm unittest now passes.

🤖 Claude Code