feat(service): support topologySpreadConstraints and priorityClassName (0.5.4)

[taha-adel]

Why

journeys-website scheduled all 5 replicas onto a single class-c-spot node, so one node event takes 100% of serving capacity. It already carried a preferred podAntiAffinity on kubernetes.io/hostname — correctly applied, selector matching the pod labels — and it still happened.

The reason is that a soft constraint needs somewhere to go. Reconstructing node state at scheduling time (10:06:21Z, mid-rollout, when the outgoing pods still held their memory):

node	free memory	can host a 1600 MiB pod?
nzh6	6664 MiB	yes — the only one
ccrg	1264 MiB	no (short by 336 MiB — its own outgoing replica held 1088 MiB)
5nxn	922 MiB	no
tc7p	912 MiB	no
litd	452 MiB	no
4gv2	324 MiB	no
v598	107 MiB	no

One feasible node, so the anti-affinity preference had nothing to act on. The pool sits at 97% of allocatable memory requests.

Why podAntiAffinity can't express the fix

requiredDuringSchedulingIgnoredDuringExecution is all-or-nothing: one node per replica, or Pending. On a pool this full that means:

• 3 of 5 replicas Pending on every rollout until the autoscaler adds nodes
• combined with a cluster-autoscaler.kubernetes.io/safe-to-evict: "false" annotation (which journeys-website now needs to stop eviction churn), it pins one node per replica against consolidation — ~$129/node/month of pool capacity the autoscaler can never reclaim
• no graceful degradation when Spot capacity is short

topologySpreadConstraints evaluates skew across the topology domains actually eligible for the pod. With maxSkew: 1, five replicas over two usable nodes is a 3/2 split — skew 1, allowed — instead of 5/0 or Pending. It spreads over whatever room exists and adapts as pool capacity changes.

What changed

Two optional scheduling fields, both rendered on the deployment.yaml and argo-rollouts.yaml pod specs, plus Chart.yaml 0.5.3 → 0.5.4.

topologySpreadConstraints

{{- with .Values.topologySpreadConstraints }}
topologySpreadConstraints:
  {{- toYaml . | nindent 8 }}
{{- end }}

priorityClassName

{{- with .Values.priorityClassName }}
priorityClassName: {{ . }}
{{- end }}

priorityClassName was the other field journeys-website wanted and this chart didn't template. Unset keeps today's behaviour exactly: no PriorityClass in this cluster sets globalDefault, so priority stays 0 — where all 302 production pods already sit. Only rudderstack-critical (1000000000) exists outside the system classes.

Two footguns are documented in values.yaml, both worth reading before anyone sets it:

• Raising priority enables preemption. The scheduler evicts lower-priority pods to make room. With every production workload at priority 0, a high value means displacing your own services, not gaining free capacity. preemptionPolicy: Never on the PriorityClass gives scheduling order without eviction.
• A wrong name fails late and quietly. Verified against the live 1.35 apiserver: a Deployment carrying a non-existent priorityClassName is accepted (--dry-run=server → configured), so ArgoCD reports Synced/Healthy. The Pod is then rejected — forbidden: no PriorityClass with name <x> was found — leaving the new ReplicaSet at 0 replicas while maxUnavailable keeps the old pods serving. It degrades silently instead of erroring. Check kubectl get priorityclasses first.

Scope: Deployment and Rollout only. cronjob.yaml and validate-job.yaml also carry pod specs and could take both fields later; left out to keep this to one reviewable surface.

Backward compatibility with 0.5.3 — verified against the whole fleet

targetRevision is pinned fleet-wide at gitops/argocd/applicationsets/prod-eg.yaml:54 (and nonprod-eg.yaml:53), so bumping to 0.5.4 upgrades every service in those ApplicationSets. Rather than argue from the diff, I rendered all of them both ways and compared.

Method: for each of the 243 service directories the ApplicationSet generators match across production, testing and integration, render with the published 0.5.3 chart and with this branch, using ArgoCD's exact valueFiles order (base/defaults.yaml, the service's values.yaml, then envs/<env>/<segment>/vars.yaml when present) and the same computed releaseName.

==================== FLEET BACKWARD-COMPAT RESULT ====================
  identical render (0.5.3 == 0.5.4) : 243
  DIFFERENT render                  : 0
  failed on BOTH (pre-existing)     : 0
  asymmetric failure (regression)   : 0
======================================================================

243/243 byte-identical. Zero diffs, zero render failures, no asymmetric behaviour.

Two structural reasons it is a guaranteed no-op, not just an empirical one:

1. The new field is [] by default and wrapped in {{- with }}, which skips empty lists — so nothing is emitted unless a service opts in.
2. No pod-template churn. service.labels in _helpers.tpl contains no helm.sh/chart label and the chart version appears nowhere in rendered output (verified: 0 occurrences of 0.5.3/0.5.4/helm.sh/chart in a full render, both versions). A chart bump that put the version in the pod template would trigger a rolling restart of ~100 workloads on sync; this one changes no pod spec at all, so ArgoCD sees no drift for any service that doesn't set the field.

Only charts/service is touched — charts/oneuptime and charts/rudderstack are unmodified.

I'd still suggest bumping nonprod-eg.yaml first and letting it sit for a sync cycle, on general principle rather than because of a specific risk.

Verification

helm lint charts/service                     -> 1 chart(s) linted, 0 failed
sh scripts/helm-docs.sh                      -> All chart README.md files are up to date
helm template (both fields set)              -> render on Deployment AND Rollout
helm template (both unset)                   -> 0 occurrences, no diff for other services
kubectl apply --dry-run=server               -> deployment.apps/journeys-website configured
fleet render 0.5.3 vs 0.5.4 (243 services)   -> 243 identical, 0 different

Docs were regenerated with local helm-docs v1.14.2, the same version as the CI container, and checked with the exact CI command rather than waiting on the job.

The Check all docs job failed on the first push for a reason worth recording: helm-docs concatenates every comment line after the # -- marker into one README table cell, so the original long rationale rendered as a single ~700-character row. Both fields now keep their rationale above the # -- line, where helm-docs ignores it — full explanation for anyone reading values.yaml, one scannable line in the README.

Follow-up (not in this PR)

1. bump targetRevision to 0.5.4 — nonprod-eg.yaml first, then prod-eg.yaml
2. gitops PR replacing journeys-website's podAntiAffinity with maxSkew: 1 / DoNotSchedule, and correcting the now-stale comment at envs/production/journeys-website/values.yaml:243 that says the chart templates neither field
3. if priorityClassName is wanted for journeys-website, a suitable PriorityClass has to be created first — rudderstack-critical at 1e9 is not it, since preempting every priority-0 production pod is not the intent

🤖 Generated with Claude Code