feat(installer): set fail-cgroupv1=false from k3s 1.31 so cgroup v1 hosts still start (backend#2422)

[LukasWodka]

Summary

Kubernetes 1.35 flipped the kubelet's failCgroupV1 default to true, so from k3s 1.35 the kubelet refuses to start on a cgroup v1 or hybrid host. That is not an exotic case for us:

• WSL2 defaults to hybrid cgroups — the Windows laptop path (AMIGO, UZ Gent).
• RHEL 8 / CentOS 7 / Ubuntu 20.04 are cgroup v1 by default — hospital Linux boxes.

k3s never sets the field, so the upstream default applies, and k3s documents none of this — an operator would see only a bare upstream kubelet message with no hint an override exists. This sets the override proactively so the refusal is never reached.

Follow-up from the backend#2422 spike. On a cgroup v2 host — every current install — it is a no-op.

Type

Feature (pre-emptive fix; inert until the k3s pin moves).

The gate is load-bearing, not a nicety

--fail-cgroupv1 was added in kubelet 1.31. Passing it to the 1.29.4 kubelet we pin today would be an unknown flag and the kubelet would not start — so an ungated version of this line breaks every install. That is why it ships with four tests rather than one, and why the flag is emitted only from 1.31.0.

Practically: this is inert on the current pin and arms itself when the pin moves, so the migration doesn't have to remember it.

Verified on a real cluster

During the spike, on v1.36.3+k3s1:

Running kubelet … --fail-cgroupv1=false --config-dir=/var/lib/rancher/k3s/agent/etc/kubelet.conf.d …
node: Ready v1.36.3+k3s1
pod: Succeeded

k3s passes it through verbatim, no parse complaint, and ps aux in the node confirms the kubelet process carries it. This settles a point two of our own investigations disagreed on: it is a real CLI flag reachable via --kubelet-arg, not a KubeletConfiguration-only field.

Why _version_lt moved to common.sh

cluster.sh needs it, and assess.sh — where it lived — is sourced conditionally by install-k8s.sh ([[ -f ]], for stale checkouts) while common.sh is not. Depending on it from cluster.sh would have failed open and silently: a missing function makes the if fail and the flag quietly not get added. One implementation, two consumers, rather than a second copy.

assess.sh's own callers are unaffected — install-k8s.sh sources common.sh first, and the bats helper's load_lib chains common.sh.

The trap this could have shipped with

_version_lt reads a leading v as a non-numeric component, i.e. 0 — so _version_lt v1.36.3 1.31.0 is TRUE and the comparison inverts. A gate written as _version_lt "$K8S_VERSION" "1.31.0" would therefore silently stop emitting the flag on exactly the versions that need it, while looking correct and passing a naive test.

Hence ${K8S_VERSION#v}, and hence a mutation anchor: dropping the strip reddens both flag-asserting tests while the two negative tests stay green. Applied and reverted the mutation to confirm the coverage is real rather than vacuous:

mutation applied → not ok 2, not ok 3 (ok 1, ok 4 still pass)
mutation reverted → all 4 ok

The trap is also pinned directly in assess.bats, documenting the behaviour rather than the intent — a gate that silently stops firing is worse than one that never worked.

Test plan

• bats scripts/tests/cluster.bats — 114/114 (4 new)
• bats scripts/tests/assess.bats — 71/71 (1 new; confirms the helper move didn't break its callers)
• shellcheck -S warning on all three touched libs — at baseline count, diffed against origin/develop rather than eyeballed:
  cluster.sh 2→2 · common.sh 15→15 · assess.sh 0→0
• scripts/gen-manifest.sh re-run and scripts/manifest.sha256 committed (Static analysis R8 gate)
• PowerShell twin updated in lockstep, using native [version] comparison; the semver strip verified against all four pin shapes (v1.29.4-k3s1, v1.31.0-k3s1, v1.36.3-k3s1, 1.35.7+k3s1)

What this does not prove

That the flag rescues an actual cgroup v1 host. That needs a WSL2 or RHEL 8 machine; it is not reproducible on macOS, where the Docker VM kernel is cgroup v2 (linuxkit.unified_cgroup_hierarchy=1). What is proven is that the flag is accepted and plumbed through, and that KEP-5573's documented purpose is exactly this override. Setting it by default is what makes the unproven half not matter: a customer never reaches the refusal.

Checklist

• Targets develop
• Self-assigned
• bats + shellcheck green, no regression vs baseline
• Manifest regenerated
• Mutation-proved
• One reviewer

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches k3d cluster create on both Unix and Windows installers. A wrong version gate or node filter would prevent kubelets from starting, though the current 1.29.4 pin leaves the flag unemitted.

Overview
Prevents kubelet refusal on cgroup v1/hybrid hosts once k3s reaches 1.35 (WSL2 hybrid, RHEL 8 / Ubuntu 20.04). Cluster create now passes --kubelet-arg=fail-cgroupv1=false@all when K8S_VERSION is latest or ≥ 1.31.

The gate is required: the flag did not exist before kubelet 1.31, so emitting it on the current v1.29.4 pin would fail every install. Scoped to @all because agents run kubelets too. Empty and digest-only pins skip.

_version_lt moves from assess.sh to common.sh so cluster.sh can use it (assess is sourced only conditionally). Callers must strip a leading v or the comparison inverts. PowerShell mirrors the same gate with a shape-checked [version] cast.

Inert on today’s pin; arms when the pin or latest default crosses 1.31/1.35.

^{Reviewed by Cursor Bugbot for commit 137899f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 10773b9. Configure here.}

[saadqbal]

Good PR — the gate reasoning, the _version_lt relocation rationale, and the leading-v mutation
anchor are all sound, and I verified the anchor does what it claims (_version_lt v1.36.3 1.31.0 is
indeed TRUE, so dropping the #v strip reddens the 1.36 test while the other three stay green).

Requesting changes on one thing: [version]$k8sSemver throws on K8S_VERSION=latest, and it
runs before the latest branch 14 lines below that exists to honour it. Confirmed under pwsh, not
inferred. The bash twin returns "don't emit" for the same input and cluster.bats pins that for the
empty case, so the suite asserts a behaviour the PowerShell half crashes on — under a comment that
says keep this in lockstep with the bash twin. This file already guards that exact pair at
:3107 and :6653.

Second comment on cluster.sh is a judgement call, not a blocker: latest/empty read as
below-1.31 and skip the flag, which is fail-closed and defensible — but that's the unpinned path,
and your own comment records it as how a client landed on v1.35.5. Worth deciding explicitly rather
than by parse accident; K3D_VERSION is pinned at v5.9.0, so the pre-1.31 hazard can't be reached
there anyway. Fine either way as long as it's stated.

[LukasWodka]

Both addressed in 137899f — and one of them was a real bug, not a nitpick.

Bugbot's High was right, and I'd copied the wrong precedent

@server:* would have left the agent kubelet refusing to start. AGENTS defaults to 1 (common.sh:849), an agent runs a kubelet, so on a cgroup v1 host --wait fails or the cluster sits half-ready — the exact refusal this change exists to prevent.

The cause is worth naming: I took the nodefilter from the --disable= args sitting directly above, which are @server:*because addon deployment is a server-only concern. A kubelet arg is not. Those two lines look identical and mean different things, so the comment now says they must not be copied from each other, and a test asserts @alland rejects @server — this is precisely the kind of line someone "tidies" into consistency with its neighbours.

Your [version] finding — confirmed, and broader than reported

You were right that it throws on latest and runs before the branch meant to honour it. Verified under pwsh, and the unguarded cast throws on three inputs:

[latest] old=THROW: RuntimeException new=emit @all
[] old=THROW: RuntimeException new=skip
[sha256:0123abc] old=THROW: RuntimeException new=skip
[v1.29.4-k3s1] old=skip new=skip
[v1.36.3-k3s1] old=emit new=emit @all

The empty case sharpens your point: cluster.bats pins empty→skip for the bash half, so the suite asserted a behaviour the PowerShell half crashed on — under a comment saying keep the two in lockstep. And a digest-only pin isn't dotted-numeric either, which :3105 already anticipates elsewhere in the file.

Fixed with this file's own pattern rather than a new one: explicit -eq "latest" first, as Test-K3sVersionDrift and the GPU gate do, then a ^\d+\.\d+ shape check before casting.

Your second comment — decided, and it changed my answer

You said it was fine either way as long as it's stated. Having to state it made me change it: latest now emits.

The reasoning, now in both twins: latest is the unsupported opt-out where k3d chooses the k3s version and we cannot read it, so the trade is a flag that is harmless from 1.31 against a refusal that is fatal from 1.35 — and latest is the very path that produced the v1.35.5 drift my own comment cites. As you noted, K3D_VERSION is pinned at v5.9.0, whose default k3s is 1.32 — above the flag's introduction, below the refusal. So emitting is safe today and becomes correct the moment k3d's default crosses 1.35, which is the direction it only ever moves.

Empty and unparseable still skip: common.sh defaults K8S_VERSION to the pin, so empty only occurs in tests.

Verification

• cluster.bats117/117 — 7 on this gate: pinned-1.29 skips, 1.31 emits, 1.36 emits (the leading-v mutation anchor), empty skips, latest emits, digest skips, and the @all nodefilter
• assess.bats71/71
• shellcheck -S warning at baseline count on all three libs, diffed against origin/develop rather than eyeballed
• bash -n clean; install-k8s.ps1 parses with 0 errors; gate logic re-run under pwsh across all six inputs
• Manifest regenerated — which also resolved its conflict with develop

bugbot run

[saadqbal]

Approving. Both of my threads are addressed and I re-ran the matrix on both engines rather than
reading the replies — the twins now agree on every input, which was the actual defect:

 pwsh bash
v1.29.4-k3s1 False false
v1.31.0-k3s1 True true
v1.35.5-k3s1 True true
v1.36.3-k3s1 True true
latest True true
(empty) False false
sha256:0123abc False false

No throws on any input, so the latest opt-out reaches its own warning instead of dying on a cast.

You changed my mind back on the second one, which is the right outcome and better reasoned than my
comment: a flag that is harmless from 1.31 against a refusal that is fatal from 1.35 is not a
symmetric trade, and latest is the path that actually produced the v1.35.5 drift. Emitting is safe
under k3d v5.9.0 and becomes correct as its default moves — the only direction it moves. Using this
file's existing -eq "latest" idiom rather than a new shape check is also the better call.

The seven tests each pin a distinct decision — 1.29 skips, 1.31 emits, 1.36 pins the #v strip,
empty skips, latest emits, a digest skips, and @all is asserted against@server:* rather than
just for itself.

And credit where it's due: Bugbot's @server:* → @all was the sharper catch and I missed it. An
agent kubelet on a cgroup v1 host would have refused to start while the server came up, so --wait
hangs or the cluster sits half-ready — the exact failure this block exists to prevent, on the WSL2
path it was written for.

One claim I could not check locally, flagged rather than doubted: that k3d v5.9.0's default k3s is
1.32. It's the load-bearing fact for emitting on latest, and it's in a comment where a future reader
will trust it — worth a link to the k3d release notes beside it.