fix(installer): enable GPU on Windows — CPU fallback + real GPU support (#616)

[shujaatTracebloc]

What & why

On Windows, a machine that has a working NVIDIA GPU still ran experiments on CPU — and silently. The training pod requested nvidia.com/gpu=1, the k3d node advertised 0 GPUs, so it sat Pending ("Insufficient nvidia.com/gpu") until the SINGLE_NODE fallback downgraded it to CPU. GPU present, GPU wasted, no signal to the user.

Root cause — an asymmetry in install-k8s.ps1: the installer requested a GPU for every job whenever it merely detected a GPU + driver, but the GPU is only provisioned in the cluster (--gpus=all, device plugin, node verify) when K3D_GPU_FLAG is set — and that's set only if the "install NVIDIA Container Toolkit into WSL" step fully succeeds, which has ~6 ways to silently bail (no WSL, distro needs first-run setup, apt/gpg blocked on a filtered network, timeouts).

Closes#616.

Layer 1 — never request a GPU the cluster can't provide ✅ (this commit)

Safe, always-correct, CPU fallback preserved:

• Gate GPU_REQUESTS/GPU_LIMITS on K3D_GPU_FLAG — the same condition that provisions the GPU, not mere detection. Empty ⇒ no GPU request ⇒ training runs on CPU. The SINGLE_NODE GPU→CPU downgrade stays as a belt-and-suspenders net.
• Loud skip: capture a specific $GPU_SKIP_REASON at each toolkit early-return (WSL not responding / Ubuntu needs first-run setup / toolkit apt blocked or timed out / verify failed), surface it in the summary (CPU (GPU detected but not enabled: <reason>)) and warn during install.
• Gate the doctor's GPU-test hint on K3D_GPU_FLAG (don't suggest a test that can't run).
• Tests: 4 new Pester tests; full suite 456 pass. manifest.sha256 regenerated.

Layer 2 — actually enable the GPU in k3d on Windows ✅ (implemented and validated on real hardware)

Makes a Windows NVIDIA GPU genuinely usable by training pods, CPU fallback preserved whenever it can't be wired up.

2a — custom k3s-CUDA node image (docker/k3s-cuda/): the stock rancher/k3s image is Alpine with no NVIDIA runtime, so GPU pods can never schedule on it. New image rebuilds the same pinned k3s on an NVIDIA CUDA Ubuntu base, installs the NVIDIA Container Toolkit, configures containerd for the nvidia runtime, and bakes in the device plugin + nvidia RuntimeClass (auto-deployed on boot). Published by the manual build-k3s-cuda workflow to ghcr.io/tracebloc/k3s-cuda:<k3s>-cuda-<base>.

2b — installer wiring (install-k8s.ps1):

• Confirm-DockerGpu — the authoritative gate: docker run --rm --gpus all nvidia/cuda:<tag> nvidia-smi. Docker Desktop uses its own WSL distro, so toolkit-in-Ubuntu was never a reliable signal — actually running a GPU container is. GPU is enabled iff this passes (so we never create a --gpus cluster that would fail, and DD-GPU works even without the user's Ubuntu toolkit). Failure ⇒ CPU (Layer 1) with a clear reason.
• Cluster create uses the custom CUDA image when GPU is enabled (stock k3s otherwise; same pinned k3s).
• Sets RUNTIME_CLASS_NAME=nvidia so jobs-manager threads runtimeClassName: nvidia into every spawned pod.
• Tests: Confirm-DockerGpu behavioral + source guards; full Pester 464 pass.

Validation status — done on real hardware (2026-08-10). Windows 11 + RTX 4050 Laptop, driver 532.10 (CUDA 12.1), Docker Desktop/WSL2. From one installer command, no registry credentials:

• ✔ GPU acceleration enabled (WSL2/CDI) → ✔ GPU verified and available → summary Mode : NVIDIA GPU
• node reports capacity/allocatable nvidia.com/gpu = 1; the CDI spec carries libdxcore (2 occurrences)
• a pod requestingnvidia.com/gpu: 1 schedules and runs CUDA: nvidia/samples:vectoradd-cuda11.2.1 → Test PASSED

Getting there required four real fixes that only surfaced on hardware, each now regression-tested: k3d ignores the image ENTRYPOINT (the CDI setup must be a /bin/k3d-entrypoint-*.sh drop-in); PowerShell 5.1 strips JSON quotes from native args (the node-capacity patch silently became value: 1 as an integer and the API server rejected it — now --patch-file); the same quoting bug in the printed doctor command; and the libdxcore injection anchor missing the generator's 4-space indent.

Still open, tracked separately: a real tracebloc training pod using the GPU needs tracebloc/client-runtime#291 (jobs-manager threads NVIDIA_VISIBLE_DEVICES into GPU pods) merged and shipped in a jobs-manager image. Everything installer- and cluster-side is complete. Note also that a fresh install currently pulls published chart 1.9.15, which predates the init-writable-data fix (chart 1.9.25) — see the promotion note in the comments.

Known limits: single GPU per node (nvidia.com/gpu=1) — correct for the single-GPU laptop scope this targets, multi-GPU is not covered. GPU on Docker Desktop cannot use the NVML device plugin at all (nvmlInit → ERROR_NOT_SUPPORTED through the paravirtualized GPU), which is why capacity is node-advertised and the GPU is injected via CDI.

🤖 Generated with Claude Code

---

Note

High Risk
Large changes to cluster creation, node status patching, and Helm GPU env wiring on the Windows install path; failures are designed to fall back to CPU but misconfiguration could still strand GPU jobs or confuse operators.

Overview
Enables real NVIDIA GPU training on Docker Desktop/WSL2 while keeping CPU fallback whenever wiring fails, and stops requesting GPUs the cluster cannot satisfy.

The custom k3s-CUDA node image switches to CDI mode (pinned NVIDIA Container Toolkit), drops the NVML device plugin (broken on WSL2), and adds a k3d entrypoint drop-in that generates the WSL CDI spec at boot, injects libdxcore.so, and re-patches nvidia.com/gpu capacity after restarts. The Helm chart gains optional env.GPU_VISIBLE_DEVICES (→ NVIDIA_VISIBLE_DEVICES in pods) for the CDI-only path; chart version 1.9.28.

install-k8s.ps1 is reworked around an authoritative docker run --gpus probe, local build or registry pull of the GPU node image (with mirror/auth), $GPU_SKIP_REASON surfaced in the summary, WSL2/CDI capacity advertising instead of the device plugin, GPU chart values gated on $K3D_GPU_FLAG, reuse/reconcile guards so CPU-only clusters do not strand jobs Pending, and single-node topology when GPU is on. check-facts and a new gpu-embed-drift.bats guard keep CUDA base tags and embedded build blobs in sync with docker/k3s-cuda/*.

^{Reviewed by Cursor Bugbot for commit 544e9d3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[shujaatTracebloc]

Note for whoever promotes this (not a change request on the PR):

This PR bumps the chart to 1.9.28. The latest published stable chart is 1.9.15, which predates the init-writable-data init container from #612 (chart 1.9.25). Since the installers pull the released chart, a fresh install on a hostPath cluster can hit unwritable ingest/training volumes until a stable chart ≥ 1.9.25 is out — data ingestion fails at the copy step.

Nothing extra to do: promoting this through the normal release train publishes the fix. Flagging it only so the ordering is understood if someone tests a fresh Windows install against the currently-published chart and sees an ingest failure.

Per-cluster workaround in the meantime, inside the node:

mkdir -p /tracebloc/<env>/data /tracebloc/<env>/logs && chmod -R 2777 /tracebloc/<env>/data /tracebloc/<env>/logs

(The confusing error that surfaced instead of a permissions message is fixed separately in tracebloc/cli#472.)

[aptracebloc]

@shujaatTracebloc (cc @saadqbal, code-owner) — Exceptionally careful work. I verified the R8/drift/chart mechanics and the core logic against a fresh clone rather than trusting the description; two items I'd like resolved before I approve, neither a code-correctness blocker.

Verified empirically

• ✅ R8 manifest re-signed correctly — shasum -c scripts/manifest.sha256 → all 18 pinned scripts OK, including the heavily-modified install-k8s.ps1. The new boot script is correctly not pinned (baked into the node image, not run by the signed bootstrap).
• ✅ check-facts drift green including the 4 new CUDA_TAG rows (ps1 CUDA_BASE_TAG, Dockerfile, build.sh, workflow — all 12.4.1-base-ubuntu22.04); check-facts.bats passes.
• ✅ Layer-1 gate is exactly right — $gpuVal/$runtimeClass/$gpuSelector default empty, set only under GPU_VENDOR=nvidia ∧ NVIDIA_DRIVER_OK ∧ K3D_GPU_FLAG≠"" (the provision condition, not detection); same triple guards Set-NodeGpuCapacity, the summary, and the doctor hint. The selector additionally rides the WSL2-only $GPU_DEVICE_SELECTOR, so Linux device-plugin nodes get an empty GPU_VISIBLE_DEVICES — consistent with client-runtime#291. Upgrade path force-clears stale values via --set-string.
• ✅ CDI boot entrypoint is genuinely fail-safe — no-op without /dev/dxg, every step || true, always exit 0, libdxcore path-probe + linker-cache fallback, parse-or-revert on the YAML edit, cdi_ok-gated detached capacity reconciler across restarts. RuntimeClass-only (no device plugin) correctly reasoned.
• ✅ Security invariants intact — runtimeClassName: nvidia only selects the containerd handler; CDI injects the device + ro,nosuid,nodev mounts; no securityContext relaxation. Training-pod isolation unchanged.
• ✅ Chart — GPU_VISIBLE_DEVICES optional string, documented, empty-by-default; helm lint clean; v1.9.28.

🟡 1. The embedded base64 has no drift guard

install-k8s.ps1 embeds the Dockerfile, RuntimeClass, and boot script as base64 (K3S_CUDA_*_B64) so the signed bootstrap can build the image locally. I decoded all three — currently byte-identical to docker/k3s-cuda/* ✓. But nothing enforces that: check-facts guards the version pins, not the full-file embedding, and Get-GpuBuildContentHash only hashes the embedded copies (local-build caching), never comparing them to the repo files. A future edit to docker/k3s-cuda/Dockerfile or the boot script that isn't re-embedded would silently diverge — workflow-published image vs installer's local build — with no CI catching it, exactly the drift class #435/#547 prevent one level up. Suggest a check-facts-style guard (or a bats test) that decodes the three blobs and diffs them against docker/k3s-cuda/*, failing CI on mismatch. In sync today, so non-blocking, but it will drift eventually.

🟡 2. Draft/validation status

The description says "kept as draft until that end-to-end validation passes" and lists "Remaining to close #616 (needs your GPU Windows box)… confirm a training pod runs on the GPU" — but the PR is not a draft. Either the live GPU-box validation was completed (please update the body), or it wasn't and this shouldn't merge yet. Mitigating: merging is bounded-safe regardless — L1 always falls back to CPU, and L2 is inert unless a GPU is present andConfirm-DockerGpu passes and the image pulls, each failure landing on CPU with a reason. So worst case on a non-GPU/failing box is "CPU as before," not breakage.

⚪ Notes

• Pester (464) not verified — no pwsh on my machine, so I couldn't run the PowerShell suite; CI must be green on it. Everything else I checked myself.
• Large PR (+2004) — cohesive (image already split to #635/#639; L1/L2 cleanly separated), so not pushing to split further, but it's over the 1000-line guideline; the install-k8s.ps1 half is the bulk.
• Set-NodeGpuCapacity and the reconciler advertise a single nvidia.com/gpu=1 — correct for the WSL2 single-GPU laptop scope; multi-GPU isn't covered (fine to note as a known limit).

Not approving yet purely on the two 🟡 questions above — the code itself is in good shape. Happy to approve once the drift-guard is added (or explicitly deferred to a follow-up) and the validation status is confirmed. Wants @saadqbal's code-owner eyes regardless (R8 + training-pod-adjacent).

🤖 Generated with Claude Code

[shujaatTracebloc]

Thanks — this is a genuinely useful review, and both 🟡 items are now handled.

🟡 1. Embedded base64 drift guard — partly a miss on my side, partly ergonomics

Half-correction: a drift guard does exist and runs in CI. install-k8s.Tests.ps1 has a Describe "Embedded GPU build inputs stay in sync with docker/k3s-cuda (#616 drift guard)" with three tests that decode each blob and compare it to the repo file — and the Pester job runs on ubuntu-latest as well as windows-latest, so an un-re-embedded edit would already have gone red. You couldn't see it because you had no pwsh, which is exactly the problem worth fixing.

Your reasoning about why nothing else covers it is precisely right and worth recording: check-facts guards the version pins, not whole-file embedding, and Get-GpuBuildContentHash only hashes the embedded copies for local-build cache-busting — it never compares them to docker/k3s-cuda/*.

So I added the bash-side mirror you asked for: scripts/tests/gpu-embed-drift.bats (0be16ab) — decodes all three blobs and diffs them against docker/k3s-cuda/*, plus a non-empty check (catches a truncated re-embed) and the drop-in shape invariant (no exec /bin/k3s, always exit 0 — the mistake that actually shipped once). CI already runs bats scripts/tests/*.bats, so it's picked up automatically, and you can now verify it locally with no pwsh.

I also checked the guard actually bites rather than assuming: appending a line to nvidia-runtimeclass.yaml without re-embedding turns test 2 red (with a diff and the exact remedy printed); restoring it returns 5/5 green. bats-hygiene passes — every assertion carries || return 1 per #527.

🟡 2. Validation status — completed; body updated

The stale "kept as draft" text was mine to fix; the description now states what was actually done. Validated on real hardware today: Windows 11 + RTX 4050 Laptop, driver 532.10, Docker Desktop/WSL2 — one installer command, no registry credentials:

• ✔ GPU acceleration enabled (WSL2/CDI) → ✔ GPU verified and available → Mode : NVIDIA GPU
• node reports nvidia.com/gpu = 1; the CDI spec carries libdxcore
• a pod requestingnvidia.com/gpu: 1 schedules and runs CUDA (vectoradd → Test PASSED)

Four real bugs only surfaced on hardware, each now regression-tested: k3d ignores the image ENTRYPOINT (the CDI setup had to become a /bin/k3d-entrypoint-*.sh drop-in); PowerShell 5.1 strips JSON quotes from native arguments, so the capacity patch silently became value: 1 as an integer and the API server rejected it (now --patch-file); the same quoting bug in the printed doctor command; and the libdxcore injection anchor assuming 2-space indent where nvidia-ctk emits 4.

Still open and tracked separately: a real training pod on the GPU needs tracebloc/client-runtime#291 (jobs-manager threads NVIDIA_VISIBLE_DEVICES) merged and shipped in a jobs-manager image. Installer/cluster side is complete.

On your ⚪ notes

• Pester is green in CI on both runners — currently 619 passing.
• Size: agreed, and I won't argue it's small. The install-k8s.ps1 half is the bulk, and roughly a third of the growth since your review is Bugbot-driven hardening rather than new capability.
• Multi-GPU: correct, and now stated as a known limit in the description. nvidia.com/gpu=1 is right for the single-GPU laptop scope; multi-GPU would need per-device CDI entries and a real device plugin, which WSL2 can't run.

Also worth flagging for your judgement: three findings in this PR recurred as classes (premature-success messages ×3, cdi list version-gating ×2) because my first fixes were too local. Each is now pinned by an invariant test rather than a patch. Per the org standard on recurring findings, I think two rules belong in .cursor/BUGBOT.md — happy to add them here or as a follow-up, your call.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0164711. Configure here.}