Uh oh!
There was an error while loading. Please reload this page.
feat(gpu): custom k3s-CUDA node image + publish workflow (#616) - #635
Merged
Conversation
Foundational build artifact for GPU-enabled edges, split into its own PR so it can land on develop and be published via CI (workflow_dispatch only dispatches from the default branch). Inert on develop — nothing references it until the installer PR (#633) wires it in. The stock rancher/k3s image is Alpine with no NVIDIA runtime, so GPU pods can never schedule on it. This image rebuilds the SAME pinned k3s on an NVIDIA CUDA Ubuntu base, installs the NVIDIA Container Toolkit, configures containerd for the `nvidia` runtime, and bakes in the device plugin + `nvidia` RuntimeClass so the node advertises nvidia.com/gpu on first boot. Based on the official k3d CUDA recipe. - docker/k3s-cuda/Dockerfile, nvidia-device-plugin-daemonset.yaml, build.sh, README.md - .github/workflows/build-k3s-cuda.yaml (manual dispatch; push=true publishes to GHCR, which also validates the Dockerfile builds) Once published (public), the installer pulls it automatically at cluster-create — the end-user runs one command; no manual build or pull. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
shujaatTracebloc added a commit
that referenced
this pull request
Aug 6, 2026
The Dockerfile + build workflow now live in #635 (targeting develop) so the image can be published via CI before this installer PR merges. #633 keeps only the installer wiring that CONSUMES the published image; nothing here changes. After #635 merges + the image is published, this branch rebases on develop to pick the build infra back up transitively. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
shujaatTracebloc
marked this pull request as ready for review
August 6, 2026 13:00
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
- workflow: pass GITHUB_TOKEN + actor via `env:` and reference quoted shell vars instead of interpolating context straight into the run line of a packages:write job (GHA injection hardening). - workflow: add `timeout-minutes: 60` so a stalled nvcr.io pull / wedged push fails fast instead of spinning to the 6h default (parity with sibling workflows). - Dockerfile: the NVIDIA keyring + apt-list curls now carry the TLS floor + bounded timeouts inline (`--tlsv1.2 --connect-timeout 30 --max-time 60`), the way scripts/install.sh spells them out (house rule; a Dockerfile can't source curl_secure()). - drift guard: the k3s pin was hardcoded in the Dockerfile ARG, build.sh, and the workflow input default with a comment CLAIMING the workflow enforced it — nothing did. Wire all three into check-facts.sh so they're verified == facts.env K8S_VERSION; a bump now fails CI (and `--write` restamps them) instead of silently deriving a GPU image tag that was never published (#547 class). check-facts.sh --check passes; shellcheck clean; workflow YAML valid. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit c666c37. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…635) Adding the GPU image's k3s pin as check-facts consumers made check-facts.bats fail — the throwaway repo the tests build only seeds scripts/, so the new Dockerfile / build.sh / workflow consumers reported "file not found" and the --check/--write tests went red. Seed those three files into setup() (matching the spec) so the drift-guard tests exercise them too. Also proves the workflow extractor picks the k3s_tag default (v… tag) and leaves cuda_tag alone. Local: check-facts.bats 14/14, bats-hygiene 18/18. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The check-facts --check gate lives in installer-tests.yaml, whose triggers were path-filtered to scripts/**. A change touching only docker/k3s-cuda/* or the build workflow never ran the gate, so a K3S_TAG edit in the GPU image could drift from facts.env and merge green — defeating the drift rows just added (Bugbot). Add docker/k3s-cuda/** and the build workflow to both the push and pull_request path filters so the gate runs on those changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
aptracebloc
approved these changes
Aug 6, 2026
Uh oh!
There was an error while loading. Please reload this page.
shujaatTracebloc added a commit
that referenced
this pull request
Aug 6, 2026
Resolve conflicts after today's develop advances (#612 chart bumps → 1.9.21, #635 build-infra + check-facts k3s-cuda consumers + installer-tests paths): - scripts/manifest.sha256: the only content conflict — regenerated against the merged tree (authoritative for the final install-k8s.ps1). - docker/k3s-cuda/* and build-k3s-cuda.yaml: took develop's versions (they live on develop now via #635; #633 had removed its copies to avoid an add/add). Verified on the merged tree: install-k8s.ps1 parses clean (GPU funcs intact), check-facts --check passes, helm-unittest 32/32, full Pester 480 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
shujaatTracebloc added a commit
that referenced
this pull request
Aug 10, 2026
…rt (#616) (#633) * fix(installer): Layer 1 — never request a GPU the cluster can't provide (#616) On Windows a machine WITH a working NVIDIA GPU still ran CPU-only: the installer requested nvidia.com/gpu for every job whenever it merely DETECTED a GPU + driver, but the GPU is only provisioned in the cluster when K3D_GPU_FLAG is set (the WSL NVIDIA-toolkit step fully succeeded). When that step didn't complete, jobs got a GPU request the node couldn't satisfy -> Pending "Insufficient nvidia.com/gpu" until the SINGLE_NODE fallback downgraded them to CPU. GPU wasted, silently. Layer 1 (safe, always-correct; CPU fallback preserved): - Gate GPU_REQUESTS/GPU_LIMITS on K3D_GPU_FLAG (the SAME condition that provisions the GPU), not on mere detection. Empty gpuVal => no GPU request => training runs on CPU. The SINGLE_NODE downgrade stays as a belt-and-suspenders net. - Make the skip LOUD: capture a specific $GPU_SKIP_REASON at each toolkit early-return (WSL not responding / Ubuntu needs first-run setup / toolkit apt blocked or timed out / verify failed), surface it in the install summary ("CPU (GPU detected but not enabled: <reason>)") and warn during install. - Gate the doctor's GPU-test hint on K3D_GPU_FLAG too (don't suggest a test that can't run). Layer 2 (actually enabling the GPU in k3d-on-Windows) follows in this PR. Tests: 4 new Pester tests (values carry no GPU request when not enabled / carry it when enabled; summary shows CPU+reason vs NVIDIA GPU). Full suite 456 pass. manifest.sha256 regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): Layer 2a — custom k3s-CUDA node image for GPU edges (#616) The stock rancher/k3s image is Alpine-based with no NVIDIA container runtime, so GPU pods can never schedule on it. Add a drop-in replacement k3s node image that rebuilds the SAME pinned k3s (K3S_TAG == installer K8S_VERSION) on an NVIDIA CUDA Ubuntu base, installs the NVIDIA Container Toolkit, configures containerd for the `nvidia` runtime, and bakes in the device plugin + `nvidia` RuntimeClass so the node advertises nvidia.com/gpu on first boot. Based on the official k3d CUDA recipe. - docker/k3s-cuda/Dockerfile (multi-stage k3s + CUDA base) - docker/k3s-cuda/nvidia-device-plugin-daemonset.yaml (device plugin + RuntimeClass, pinned v0.14.5) - docker/k3s-cuda/build.sh (build/push, tag encodes k3s + CUDA versions) - docker/k3s-cuda/README.md - .github/workflows/build-k3s-cuda.yaml (manual dispatch; push=true publishes to GHCR) Image tag encodes both the k3s pin and CUDA base so a new k8s pin can't silently reuse a stale GPU image. The installer wiring to actually USE this image (cluster --image swap, GPU capability probe, RUNTIME_CLASS_NAME=nvidia) lands next in this PR; CPU fallback (Layer 1) always remains. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): Layer 2b — wire the installer to the custom k3s-CUDA image (#616) Make a Windows NVIDIA GPU genuinely usable by training pods, with CPU fallback whenever it can't be wired up. - Authoritative GPU gate: add Confirm-DockerGpu, which runs `docker run --rm --gpus all nvidia/cuda:<tag> nvidia-smi`. Docker Desktop uses its OWN WSL distro, so the toolkit-in-Ubuntu step was never a reliable signal; actually running a GPU container is. Enable GPU iff the probe passes -> we never create a `--gpus` cluster that would fail, and DD-GPU works even without the user's Ubuntu toolkit. Probe failure => K3D_GPU_FLAG cleared + a clear reason => CPU (Layer 1). - Cluster create: use the custom k3s-CUDA image ($K3S_CUDA_IMAGE, env-overridable via TRACEBLOC_K3S_CUDA_IMAGE, default ghcr.io/tracebloc/k3s-cuda:<K8S_VERSION>-cuda-<base>) when GPU is enabled; stock rancher/k3s otherwise. Same pinned k3s either way. - Set RUNTIME_CLASS_NAME=nvidia in values when GPU is enabled, so jobs-manager threads runtimeClassName: nvidia into every spawned pod (the RuntimeClass is baked into the CUDA image). Empty otherwise. Tests: Confirm-DockerGpu behavioral tests (pass/fail/no-GPU) + source guards for the image swap, probe gating, and RuntimeClass; existing GPU values tests extended for RUNTIME_CLASS_NAME. Full Pester suite 464 pass. manifest.sha256 regenerated. NOTE: end-to-end GPU execution needs validation on a real GPU Windows box (build + publish the image via the build-k3s-cuda workflow, then re-install). CPU path is unaffected and fully covered. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): make the GPU node image mirror-aware (one-command, air-gap) (#616) The one installer command must do everything with no external steps — the user never builds or pulls the GPU image by hand; the installer pulls it automatically at cluster-create. Re-home the default k3s-CUDA image onto TRACEBLOC_IMAGE_REGISTRY when a private mirror is configured (#585), so the single command also works on a restricted/air-gapped network, same as every other image. Explicit TRACEBLOC_K3S_CUDA_IMAGE still overrides. Source guard updated. Pester 464 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(installer): move k3s-CUDA image build infra to its own PR (#635) The Dockerfile + build workflow now live in #635 (targeting develop) so the image can be published via CI before this installer PR merges. #633 keeps only the installer wiring that CONSUMES the published image; nothing here changes. After #635 merges + the image is published, this branch rebases on develop to pick the build infra back up transitively. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): pull the private GPU node image with creds — keep it private (#616) We keep ghcr.io/tracebloc/k3s-cuda PRIVATE rather than public. So the installer must authenticate to pull it — the end user still runs ONE command; the creds come from env, not a separate docker login. - Confirm-GpuImagePullable: derives the registry host from $K3S_CUDA_IMAGE, `docker login`s with TRACEBLOC_REGISTRY_USERNAME/PASSWORD (the same vars the mirror uses, #585) when set, then `docker pull`s the image — which both VERIFIES access and PRE-LOADS it so k3d cluster-create reuses the local copy (no second pull). - The GPU gate now requires BOTH the passthrough probe AND a pullable image: `if ((Confirm-DockerGpu) -and (Confirm-GpuImagePullable))`. Either failure => clean CPU fallback with a specific reason (private-registry hint names the exact env vars to set; cred-set-but-failed says to check read access) instead of a cluster-create that dies pulling an unauthorized image. Tests: 4 Confirm-GpuImagePullable cases (login+pull ok / cred fail / no-cred hint / gate source-guard); updated the stale probe-gate guard. Full Pester 468 pass. manifest regen. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): bound the GPU docker calls with a timeout (Bugbot) (#616) The new `docker run` (GPU probe), `docker login`, and `docker pull` calls were unbounded — a wedged Docker daemon, registry, or proxy could hang Step 2 forever instead of falling back to CPU (installer external-call timeout rule). Route all three through a new Invoke-DockerCli helper that runs docker in a bounded background job (Wait-JobWithProgress deadline); on timeout it kills the job and returns Code=124 so callers fall back to CPU cleanly. Timeouts: probe 180s, login 60s, pull 900s (the GB CUDA image on a slow/proxied network — generous but finite). Login stdin (the token) is passed in-memory via the arg hashtable, never to disk, argv, or logs. Tests: rewired Confirm-DockerGpu / Confirm-GpuImagePullable tests to mock Invoke-DockerCli (regular function — mockable, unlike a job'd docker), added timeout-path (Code 124 -> CPU) cases + a source guard that every GPU docker call is bounded. Full Pester 482 pass. manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): drift-guard the CUDA tag via facts.env (Bugbot) (#616) CUDA_BASE_TAG (the CUDA base that derives the pulled GPU image tag) was hardcoded in four places — install-k8s.ps1, docker/k3s-cuda/Dockerfile ARG, build.sh, and the build-k3s-cuda workflow default — outside facts.env / check-facts, so a CUDA bump in one spot could leave the installer pulling a GPU image tag that was never built (#547-class drift, same as the k3s pin). Add CUDA_TAG=12.4.1-base-ubuntu22.04 to facts.env and wire all four consumers into check-facts.sh (verified == facts.env; --write restamps them). The workflow extractor distinguishes the cuda_tag default (…-ubuntu…) from the k3s_tag default (vX…k3s). check-facts.bats fixture seeds the two new consumers. check-facts.sh --check passes (15 consumers), check-facts.bats 14/14, shellcheck clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): kill docker on timeout — no orphaned native process (Bugbot) (#616) Invoke-DockerCli used Start-Job; Stop-Job stops the PS job but can orphan the native docker.exe it spawned, so a timed-out run/login/pull could keep going after the installer fell back to CPU. Run docker as a direct child Process instead and Kill() it on timeout, so the deadline actually terminates docker. 5.1-safe (.NET Framework Process API); args have no spaces (flags + image tags) so a plain join is safe; login stdin is written in-memory, never to disk/argv/logs. Callers are unchanged (still mock Invoke-DockerCli); updated the source guard to assert WaitForExit(timeout) + Kill(). Full Pester 482 pass. manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): accurate GPU-skip reasons + drift-check the GPU image (Bugbot) (#633) Two Bugbot findings: - GPU_SKIP_REASON now reflects the ACTUAL failure. A probe/pull timeout (Code 124) reports "timed out", not a GPU-unavailable or credentials error; a real GPU passthrough failure reports the Docker Desktop hint, not stale Ubuntu guidance. - Test-K3sVersionDrift now recognizes the GPU node image (…/k3s-cuda:<k3s>-cuda-<base>), extracting the k3s pin from before "-cuda-", so a GPU cluster no longer silently escapes the version-drift check after a pin bump (it only parsed rancher/k3s: before). Tests: reason assertions on the timeout/non-zero paths + a drift source guard for the GPU image. Full Pester 483 pass. manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): single-node GPU cluster so one card isn't double-counted (Bugbot) (#616) k3d's --gpus=all exposes the SAME host GPU to every node container, and the baked-in device-plugin DaemonSet registers it once per node. A default server+agent cluster therefore advertises nvidia.com/gpu=1 on BOTH nodes -- 2 allocatable for 1 physical card -- and Kubernetes can schedule two jobs onto the same device. Extra k3d nodes live on the same Docker host and all see the same card, so multi-node can never add real GPUs; it only double-counts. When the GPU is enabled, collapse to a single node (agents=0) so the card is advertised exactly once. An explicit user AGENTS is overridden loudly, not silently. CPU installs are unchanged (topology only shifts inside the K3D_GPU_FLAG="--gpus=all" branch). +1 source-guard test; full Pester 484 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): reused CPU cluster can't silently get GPU values (Bugbot) (#616) The GPU gate enables --gpus=all + GPU chart values in Step 2, before New-K3dCluster runs. On a re-install that REUSES an existing cluster (rather than recreating it), a cluster first built in CPU mode keeps its stock rancher/k3s node -- no NVIDIA runtime, advertises 0 GPUs, no `nvidia` RuntimeClass -- and k3s's node image is fixed at create time. Writing GPU values against it would strand every experiment Pending: exactly the #616 failure this PR removes. Reconcile in the reuse path: Confirm-ReusedClusterGpuCapable inspects the reused server node's image (bounded docker inspect, mirrors Test-K3sVersionDrift) and, when GPU was requested but the node isn't the custom k3s-CUDA image, disables GPU for this run (CPU fallback stays safe) and tells the user to recreate the cluster to enable GPU. The healthy fast-path already exits before the GPU gate, so it can't wrongly enable GPU. Pure Test-NodeImageGpuCapable classifier is unit-tested (stock/CUDA/mirror/ empty); the bounded inspect + clear-flag + call-site are source-guarded. Full Pester 492 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): build the GPU node image locally so GPU installs need no login (#616) Restores the one-command promise for GPU. The GPU node image was published private, so enabling GPU forced the user to supply a registry token -- exactly the friction #612 (CPU) never had, because every image it pulls is public. Default GPU path is now a LOCAL build: on an NVIDIA machine the installer runs `docker build` from PUBLIC bases only (rancher/k3s + NVIDIA's public nvcr.io CUDA base + the public NVIDIA container toolkit), tags it locally, and creates the cluster with that image -- no registry login, no private package, one command. Build-GpuNodeImage is idempotent (reuses an already-built image), bounded with a visible progress bar (Wait-ProcessWithDeadline), sanity-checks that the built image runs k3s, and on any failure falls back to CPU with a clear reason. The Dockerfile + device-plugin manifest are embedded (self-contained verified artifact) and a Pester drift guard keeps them byte-identical to docker/k3s-cuda/*. An explicit prebuilt image / mirror (TRACEBLOC_K3S_CUDA_IMAGE or TRACEBLOC_IMAGE_REGISTRY, i.e. air-gap tenants) still PULLS instead of building. CPU is untouched: the whole block is behind the NVIDIA gate + the docker-run probe, so a CPU machine builds nothing and gets stock rancher/k3s exactly as before. Windows-installer only; staging/prod (Linux + chart) are not affected. Full Pester 499 pass; check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): 5 GPU edge cases from Bugbot (latest, reuse, preflight, fallback) (#616) Round-3 Bugbot findings on the GPU installer, all CPU-safe: - Healthy reinstalls skip the GPU fix (Med): the completed+healthy fast path exited before any GPU reconciliation, so a cluster whose values request GPU on a CPU-only node kept stranding jobs while looking healthy. Add Test-HealthyClusterGpuConsistent to the fast path -- warns + recreate remedy. - K8S_VERSION=latest bypasses the CUDA image (Med): with latest, cluster-create adds no --image (stock node) but GPU stayed enabled -> stranded jobs. Refuse GPU for latest/empty K8S_VERSION at the gate (CPU fallback + reason). - Custom GPU image overrides fail reuse (Med): the reuse check only matched `k3s-cuda:`, downgrading a renamed/digest mirror image. Test-NodeImageGpuCapable now also accepts an exact match against the configured $K3S_CUDA_IMAGE. - Optional GPU build could abort the install (Med): Build-GpuNodeImage created the build-context dir OUTSIDE its try, so a temp/AV/disk error hit the fatal trap. Moved dir creation inside try + added a catch -> CPU fallback. - nvcr.io missing from preflight (Low): Find-Gpu now runs before Test-Preflight, which probes nvcr.io + nvidia.github.io when an NVIDIA GPU is present and we'll build. SOFT (warn, not hard-fail) since GPU degrades to CPU. +10 tests; full Pester 509 pass; check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): base64-embed the GPU build inputs so host lints don't trip (CI green) The raw here-string embed of docker/k3s-cuda/* broke two host-side guards that scan install-k8s.ps1: - check-style.sh flagged `apt-get install ... curl` (curl as a PACKAGE name) as a bare-curl call. - install.Tests.ps1's "no non-ASCII in string literals" test flagged the em-dashes in the embedded Dockerfile/manifest comments (an existing test, not to be rewritten). Embed the two files as base64 instead (pure ASCII, no `curl` token), decode them to the build context at build time. Same self-contained + drift-guarded property (the Pester drift test now decodes and compares to docker/k3s-cuda/*), with nothing for the container-vs-host lints to catch. No behaviour change to the build itself. Ran the FULL scripts/tests dir this time (both install.Tests.ps1 and install-k8s.Tests.ps1): 541 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): mirror-aware GPU probe + preflight the configured GPU registry (Bugbot) (#616) Two air-gap/mirror findings: - Mirror GPU path still hit Docker Hub (Med): the GPU passthrough probe pulled nvidia/cuda from Docker Hub regardless of the mirror. Add $CUDA_PROBE_IMAGE, re-homed onto TRACEBLOC_IMAGE_REGISTRY when set, and use it in Confirm-DockerGpu -- so a mirrored/air-gapped GPU install doesn't fall back to CPU on a blocked Docker Hub. (The device-plugin image baked into a PREBUILT node image is the mirror operator's build-time concern; the installer only pulls that image.) - Custom GPU registry bypassed preflight (Low): the new nvcr.io probes were added only on the build path, but the pull path contacts the configured registry. Add an else-branch that probes the $K3S_CUDA_IMAGE registry host (when it's a real host, not a bare Docker Hub repo) so an unreachable custom/mirror registry is surfaced at preflight, not at pull time. All GPU probes stay SOFT (warn -> CPU fallback), never hard-failing a CPU-capable install. Full scripts/tests: 543 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): regenerate embedded GPU build inputs from the merged Dockerfile (#616) Merged develop, which now carries #639's k3s-CUDA Dockerfile fix. PR CI builds the branch MERGED with develop, so the drift test compared the embedded base64 (generated from #633's older Dockerfile) against develop's newer one and failed. Regenerated the base64 embed from the current docker/k3s-cuda/Dockerfile + device-plugin manifest so the embed matches the merge tree. Full scripts/tests: 543 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): GPU registry login before the probe + correct Docker Hub host (Bugbot) (#616) Two mirror/auth findings from the mirror-aware probe change: - Private mirror probe ran before login (Med): with an authenticated TRACEBLOC_IMAGE_REGISTRY, Confirm-DockerGpu pulled the mirror-hosted CUDA probe image BEFORE Confirm-GpuImagePullable did `docker login`, so a credentialed install was rejected and fell back to CPU. Extracted Connect-GpuRegistry and call it in the gate BEFORE the probe (and reuse it inside Confirm-GpuImagePullable). - Docker Hub overrides logged into the wrong host (Low): for a bare `owner/image` ref, the login host was `owner` instead of Docker Hub. Added the pure Get-RegistryHost: a segment is a registry only if it has a '.'/':' or is 'localhost'; otherwise login targets docker.io. Full scripts/tests: 550 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): reconcile GPU on adopted-reuse + re-verify a reused GPU image (Bugbot) (#616) - Adopted releases retained stale GPU requests (Med): the adopted-reuse path uses helm --reuse-values, so an older release's GPU_REQUESTS/GPU_LIMITS survived even after cluster reconciliation cleared K3D_GPU_FLAG -> stranded jobs. Moved the GPU value decision before the adopted/fresh split and pass the three GPU env keys as --set-string on the adopted upgrade, so both paths reconcile GPU to this run. - Failed GPU images were reused unchecked (Med): Build-GpuNodeImage treated any existing tag as ready. A build that completed but failed the k3s sanity check left a broken image behind that the next run would reuse. Extracted Test-GpuImageRunsK3s and run it on the reuse path too; a broken existing image is rebuilt, not reused. Full scripts/tests: 554 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): node is authoritative for GPU + login to all GPU registries (Bugbot) (#616) - Mirrored installs could keep GPU on with a dead device plugin (Med): the device-plugin image is baked as nvcr.io/... and on a blocked mirror/air-gap network the DaemonSet never becomes ready, so the node advertises 0 GPUs while GPU requests stayed active -> stranded jobs. Confirm-GpuNode now makes the NODE authoritative: after the allocatable-GPU wait, a 0 count clears K3D_GPU_FLAG (before Install-ClientHelm writes values) -> CPU fallback with a clear reason. - Combined registry overrides logged into the wrong host (Med): with TRACEBLOC_K3S_CUDA_IMAGE and TRACEBLOC_IMAGE_REGISTRY on different hosts, Connect-GpuRegistry logged into the node-image host while the probe pulled from the mirror host -> probe rejected, GPU needlessly disabled. It now logs into every DISTINCT host across the node image and the probe image. (The earlier "Static analysis" red was a transient PSGallery download corruption installing PSScriptAnalyzer -- infra flake, cleared by re-running CI; no code cause.) Full scripts/tests: 557 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): sanity-check pulled GPU images + full pull-path preflight (Bugbot) (#616) - Pulled node images bypassed the sanity check (Med): Confirm-GpuImagePullable accepted any successful pull, so a mis-tagged/broken mirror image enabled GPU and then aborted k3d cluster-create. It now runs Test-GpuImageRunsK3s on the pulled image (symmetric with the local build) -> CPU fallback on a broken image. - GPU preflight missed required hosts (Med): the pull-path preflight probed only $K3S_CUDA_IMAGE's host. nvcr.io (the baked device-plugin image, pulled at runtime on BOTH paths) is now probed whenever GPU is enabled, and the pull path probes every DISTINCT host across the node image AND the probe image (they can differ when both overrides are set). All SOFT (warn -> CPU fallback), never a hard fail. Full scripts/tests: 559 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): a null docker-build exit code is not a failure (Bugbot) (#616) With redirected stdout/stderr, $proc.ExitCode can remain $null after the process exits (even though Wait-ProcessWithDeadline calls WaitForExit()), so `$proc.ExitCode -ne 0` misclassified a SUCCESSFUL GPU build as failed and silently dropped GPU. Mirror the #611 k3d-create idiom: fail only on a CONFIRMED non-zero exit; a null code defers to the k3s sanity check (Test-GpuImageRunsK3s), which is the authoritative "did the build produce a working image" success marker. A null exit + working image now enables GPU; a null exit + broken image still falls back to CPU. Full scripts/tests: 561 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): GPU node-verification + fallback correctness (Bugbot) (#616) Four findings surfaced on the develop-merge review: - GPU verification always read zero (High): Confirm-GpuNode parsed the whole allocatable map (`jsonpath='{...allocatable}'` renders Go's map[...] with no JSON quotes), so the "nvidia.com/gpu" regex never matched -> every healthy GPU read as 0. Combined with the new 0-count fallback that reverted ALL GPU installs to CPU. Fixed: target the scalar field directly (jsonpath '...allocatable.nvidia\.com/gpu'). - Plugin failure left GPU requests enabled (High): when Install-GpuDevicePlugin returned false, K3D_GPU_FLAG stayed set and Confirm-GpuNode was skipped, so the chart still requested GPUs -> stranded jobs. Now a real plugin failure (flag still set) clears the flag -> CPU fallback. - K8S_VERSION=latest retained the flag (Med): the latest branch recorded a reason but didn't clear K3D_GPU_FLAG that Install-NvidiaContainerToolkit may have set -> a stock 'latest' cluster could get --gpus=all without the CUDA image. Now cleared. - Fast-path read stale GPU values (Low): Test-HealthyClusterGpuConsistent read local values.yaml, which is only clientId-healed on the adopted path. Now reads the LIVE Helm release values (helm get values) instead. Full scripts/tests: 564 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): disable NVIDIA_REQUIRE_CUDA so an older-but-valid driver enables GPU (#616) Real-box test (RTX 4050, driver 532.10 = CUDA 12.1) showed GPU falling back to CPU. Root cause: the CUDA base image bakes NVIDIA_REQUIRE_CUDA=cuda>=12.4, and the NVIDIA container runtime REFUSES to start the container when the driver is older ("unsatisfied condition: cuda>=12.4"). So a perfectly good GPU on a slightly older driver read as "Docker can't expose the GPU" and dropped to CPU. The probe and the k3s node run nvidia-smi / k3s, NOT CUDA workloads -- the real CUDA/driver compatibility for training is enforced per-pod by each training image -- so this requirement gate is meaningless for us. Disable it: - Confirm-DockerGpu probe: docker run ... -e NVIDIA_DISABLE_REQUIRE=1 - node image: ENV NVIDIA_DISABLE_REQUIRE=1 (so the k3s node boots under --gpus on an older driver); base64 embed regenerated. Now GPU enables on any driver that can actually pass the GPU through, regardless of the base image's CUDA version. Full scripts/tests: 566 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): key GPU-image reuse on build-input content hash, not just the tag (Bugbot) (#616) Adding NVIDIA_DISABLE_REQUIRE didn't change the image tag, and Build-GpuNodeImage reused any cached image that merely printed a k3s version -- so a stale pre-fix image would be reused and still fail cluster-create on an older driver. Stamp a short content hash of the build inputs (embedded Dockerfile + device-plugin manifest) as a `tracebloc.k3s-cuda-content` label at build time, and on reuse require that label to match the CURRENT hash (plus the k3s sanity check). Any change to the build inputs now busts the cache -> a stale image is rebuilt, not reused. Generalizes beyond this one fix to any future Dockerfile change. Full scripts/tests: 568 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): fast-path retries GPU + image sanity check exercises --gpus (Bugbot) (#616) - Healthy fast path prevented GPU retries (Med): a completed CPU-fallback install exited "nothing to do" before Find-Gpu/probe, so re-running after fixing Docker/driver/registry could never enable GPU -- contradicting the fallback guidance. Find-Gpu now runs before the fast path, and when an NVIDIA GPU is present but the running cluster is CPU-only (Test-RunningClusterGpuCapable) the fast path falls through to the full walk to retry GPU instead of exiting. - Image sanity check skipped GPU startup (Med): Test-GpuImageRunsK3s ran without --gpus, so a stale mirror image lacking NVIDIA_DISABLE_REQUIRE passed the check and then failed k3d cluster-create on an older driver (aborting instead of CPU fallback). It now runs WITH --gpus (no bypass), exercising the exact requirement gate cluster-create hits -> a stale image drops to CPU cleanly. Full scripts/tests: 571 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): fast-path GPU-live check uses allocatable GPU + bound nvidia-smi (Bugbot) (#616) - Fast path mistook a CUDA image for a working GPU (Med): Test-RunningClusterGpuCapable keyed on the node IMAGE name, but a CUDA node with a failed device plugin has 0 GPUs -- so a CPU-fallback re-run would wrongly take the fast path and never retry. It now checks the node's LIVE allocatable nvidia.com/gpu (the same authoritative signal Confirm-GpuNode uses), bounded with --request-timeout. - Unbounded nvidia-smi in Find-Gpu (Med): Find-Gpu now runs before the fast path, and its driver/VRAM nvidia-smi probes had no deadline, so a wedged driver could hang every re-run. Extracted Invoke-BoundedProcess (generic bounded+killable child; Invoke-DockerCli is now a thin wrapper over it) and Confirm-NvidiaDriver uses it for nvidia-smi (15s). Full scripts/tests: 572 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): fast path reconciles GPU both directions -- enable AND CPU-drift (Bugbot) (#616) The prior fix only handled "GPU present but node CPU-only -> retry". The reverse also leaked: when the node now advertises a GPU (e.g. a delayed device-plugin recovery) but the live release still requests CPU, the fast path exited "nothing to do" and training silently stayed on CPU. The fast path now shortcuts ONLY when GPU is FULLY consistent -- the node advertises a GPU AND the live release requests one (Test-RunningClusterGpuCapable + Test-LiveReleaseRequestsGpu). Any other combo on a GPU machine falls through to the full walk, which reconciles GPU_REQUESTS in either direction. Extracted the live-release GPU check into Test-LiveReleaseRequestsGpu (shared with Test-HealthyClusterGpuConsistent). Full scripts/tests: 572 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gpu): real GPU on Windows/WSL2 via CDI (validated on hardware) (#616) Validated end-to-end on a live RTX 4050 laptop (driver 532.10, Docker Desktop/WSL2): a pod that REQUESTS nvidia.com/gpu now schedules AND runs CUDA compute (vectorAdd "Test PASSED"). Previously GPU always fell back to CPU on Windows. Why the old approach couldn't work, and what replaces it: * The NVIDIA k8s DEVICE PLUGIN cannot work on Docker Desktop/WSL2: the GPU is paravirtualized (/dev/dxg + a WSL driver store), so nvmlInit() returns ERROR_NOT_SUPPORTED and the plugin registers 0 GPUs. Because it OWNS the nvidia.com/gpu extended resource it also pins the node at 0, stranding jobs. -> the node image no longer ships it (RuntimeClass only), and the installer advertises nvidia.com/gpu itself via a node-status patch (Set-NodeGpuCapacity). * CUDA itself works fine through CDI. The node image now runs nvidia-container-runtime in CDI mode and generates the WSL CDI spec AT BOOT (tracebloc-cdi-boot.sh) -- it must be at boot because the driver-store path contains a per-machine hash. * Critical gap found: `nvidia-ctk cdi generate --mode=wsl` OMITS libdxcore.so (it lives in the standard lib path, not the driver store). Without it libcuda loads but can't reach /dev/dxg, surfacing as a misleading "CUDA driver version is insufficient for CUDA runtime version". The boot script injects it. CPU and Linux are unaffected: the boot script is a strict no-op without /dev/dxg, the installer only takes the CDI path when /dev/dxg exists in the node (otherwise the standard device-plugin path is unchanged), and every step failure-falls back to CPU. Follow-up (separate repo/PR): jobs-manager must set NVIDIA_VISIBLE_DEVICES=nvidia.com/gpu=all on GPU training pods. +8 tests; full scripts/tests 580 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): node self-heals GPU capacity + verify the CDI spec (Bugbot) (#616) - GPU capacity lost after node restart (HIGH): nvidia.com/gpu was advertised by a one-shot install-time status patch, but a manually patched extended resource is NOT durable -- the kubelet re-reports node status on start, so a Docker Desktop or Windows restart dropped it while the chart still requested a GPU, stranding every job Pending until the installer was re-run. There's no device plugin to own the resource on WSL2, so the NODE now re-asserts it: the boot script (the entrypoint, so it runs on every start) backgrounds a reconciler that waits for the local API and re-patches whenever the capacity is missing/0. A reboot self-heals with no user action. Interval override: TRACEBLOC_GPU_RECONCILE_SECS. - CDI success wasn't verified (MED): the WSL2 path printed "GPU acceleration enabled" after only the capacity patch. Since the boot script guards every step with `|| true`, a failed `nvidia-ctk cdi generate` would leave us advertising a GPU pods can't use -- jobs schedule then fail CUDA with no cluster-level signal. Now the installer verifies /etc/cdi/nvidia.yaml exists and is non-empty in the node before claiming success, else CPU fallback with a clear reason. Both stay CPU-safe: the reconciler is backgrounded + fully guarded (can never delay or block k3s) and only runs when /dev/dxg exists; the spec check failure-falls back to CPU. +2 tests; full scripts/tests 582 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gpu): chart wiring for the CDI device selector + path-correct GPU guidance (#616) Completes the GPU-on-Windows chain: the installer now tells the chart to thread the CDI device selector into GPU training pods, which jobs-manager consumes as NVIDIA_VISIBLE_DEVICES (tracebloc/client-runtime#291). Without this the node advertises a GPU and pods schedule, but CUDA fails inside them. Chart: documented env.GPU_VISIBLE_DEVICES in values.yaml + values.schema.json. No template change needed -- jobs-manager-deployment's generic env pass-through already forwards it to both containers (verified with `helm template`: present when set, absent when unset/empty). Chart 1.9.27 -> 1.9.28 (version + appVersion in lockstep). Installer: $GPU_DEVICE_SELECTOR is set ONLY on the WSL2/CDI path and written as env.GPU_VISIBLE_DEVICES (also forced via --set-string on the adopted-reuse reconcile, so a stale value can't survive). Empty on a normal device-plugin (Linux) node, where the plugin owns NVIDIA_VISIBLE_DEVICES and sets concrete GPU UUIDs -- forcing a CDI selector there would break device resolution. Also fixes two misleading-guidance findings (Bugbot): - The 0-GPU skip reason always blamed the NVIDIA device plugin and told operators to mirror nvcr.io/nvidia/k8s-device-plugin. On the WSL2/CDI path there IS no device plugin, so that was a dead end. Now branches per path. - The doctor's GPU smoke-test command omitted runtimeClassName: nvidia (pods only get the GPU under it), so running it on a WORKING cluster looked like failure. It now carries the RuntimeClass, and on WSL2/CDI suggests a CUDA workload instead of nvidia-smi -- verified on real hardware, nvidia-smi fails in a pod on WSL2 (NVML unsupported) even when CUDA compute works. +4 tests; full scripts/tests 586 pass / 0 fail; check-style, check-facts, helm lint green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): remove a leftover device plugin, correct the CDI cause, collapse SERVERS (Bugbot) (#616) - CDI path left the device plugin fighting (HIGH): a device plugin OWNS the nvidia.com/gpu extended resource and on WSL2 re-reports 0 on every sync, so a DaemonSet left behind by an older install (or by a run where the /dev/dxg probe transiently missed and we took the plugin path) would overwrite the capacity patch and keep GPU disabled across ALL re-runs. The CDI path now deletes it first (--ignore-not-found, so the normal case is a clean no-op) before advertising. - Wrong CPU-fallback reason on CDI (MED): when Set-NodeGpuCapacity failed, no GPU_SKIP_REASON was set, so the caller's generic fallback filled in a device-plugin failure -- the wrong cause on a path that never uses the plugin. Set it here. - GPU mode ignored SERVERS (MED): AGENTS was collapsed to 0 but SERVERS was left alone, and EVERY server node runs the boot reconciler and advertises nvidia.com/gpu=1 for the SAME physical card -- so SERVERS>1 offered N GPUs for one device. GPU mode now forces servers=1 too, overriding an explicit value loudly. +3 tests; full scripts/tests 589 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): ship CDI setup as a k3d entrypoint DROP-IN -- the image ENTRYPOINT never ran (#616) Root cause of "GPU couldn't be wired into the cluster (CDI spec missing)" on a real Windows box. k3d does NOT use the node image's ENTRYPOINT: it replaces it with its own /bin/k3d-entrypoint.sh, which runs every /bin/k3d-entrypoint-*.sh drop-in and then execs k3s. Confirmed on the live node: Entrypoint = ["/bin/k3d-entrypoint.sh"] /bin/k3d-entrypoint-{cgroupv2,dns,mounts}.sh <- k3d's own drop-ins /usr/local/bin/tracebloc-cdi-boot.sh <- ours, never invoked /etc/cdi <- missing (spec never generated) So the CDI spec was never created and the installer correctly fell back to CPU (the verification added in c37e133 is what surfaced this instead of silently "succeeding"). Fix: install the script as /bin/k3d-entrypoint-tracebloc-cdi.sh -- k3d's documented extension point -- and restore the stock ENTRYPOINT ["/bin/k3s"]. As a drop-in it must RETURN rather than exec k3s (k3d does that), and must always exit 0: k3d runs drop-ins with `|| exit 1`, so a non-zero exit would abort the whole node, and GPU is optional. The capacity reconciler is now fully detached (</dev/null, output to /dev/null) since the drop-in exits immediately after forking it. Also corrected the Dockerfile header, which still claimed the image bakes in the device plugin (it ships only the RuntimeClass now). +2 regression tests (drop-in path + stock ENTRYPOINT; returns and never execs k3s); full scripts/tests 591 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): pass the node capacity patch via --patch-file (PS 5.1 ate the JSON quotes) (#616) Root cause of "Couldn't advertise GPU capacity on the node" on a real Windows box. Windows PowerShell 5.1 does not preserve embedded double quotes when it builds a native command line, so kubectl patch node N --type=json -p '[{"op":"add","path":...,"value":"1"}]' reached kubectl as [{op:add,...}] -> invalid JSON -> the patch failed on EVERY run. The same command works by hand only because each quote is escaped as \" there. (This is why the CDI spec now generates fine but capacity never appeared.) Fix: write the patch to a temp file and use --patch-file, which sidesteps the shell entirely. UTF8 without BOM (a BOM breaks kubectl's JSON parse). Also retried 6x/5s: the node object can still be settling right after cluster-create, and a transient 404 there would drop an otherwise-working GPU to CPU for the whole run. Temp file always cleaned up. +1 regression test (asserts --patch-file, no inline -p, BOM-less encoding, retry loop); full scripts/tests 592 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): print the doctor's GPU test with escaped quotes so it can actually be pasted (#616) Same PowerShell quote-stripping class as the capacity patch, now in user-facing guidance: the doctor printed `--overrides='{"spec":...}'`, which dies with "error: Invalid JSON Patch" when pasted into Windows PowerShell (confirmed on a live box). A suggested command that can't be run is worse than none, so the JSON is now emitted with every quote escaped as \" -- the form proven to work by hand. Verified the rendered string is paste-able: --overrides='{\"spec\":{\"runtimeClassName\":\"nvidia\", ... }}' full scripts/tests 592 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): libdxcore injection missed the generator's indentation (real CUDA failure) (#616) Root cause of "CUDA driver version is insufficient" inside GPU pods on a real box. Confirmed live: the CDI spec existed and parsed (`nvidia-ctk cdi list` -> 1 device), but `grep libdxcore /etc/cdi/nvidia.yaml` found NOTHING -- the injection never happened. The awk anchor was `^ mounts:$` (exactly two spaces), while nvidia-ctk marshals YAML with yaml.v3 at FOUR spaces, so it never matched. Without libdxcore, libcuda loads but can't reach /dev/dxg -> that misleading driver error, on a node that advertises a healthy GPU. Fixes: - Indent-agnostic anchor (`^[[:space:]]*mounts:[[:space:]]*$`) and the inserted item now MIRRORS the generator's own first-item indentation instead of hardcoding it. Hardcoding was doubly wrong: YAML forbids mixing indents in one list, so a fixed 4-space item next to 2-space items would have made the whole spec unparseable (CDI then injects nothing -- same symptom, different cause). Verified both 2- and 4-space specs parse with libdxcore present, and that the old awk produced invalid YAML on a 2-space spec. - The edit is only adopted if `nvidia-ctk cdi list` still parses the result; otherwise the original spec is restored (GPU without libdxcore beats a silently disabled GPU). - The installer now verifies libdxcore is IN the spec (not merely that a spec exists) and falls back to CPU with a specific reason if not -- so this class of silent miss surfaces at install time instead of as a confusing CUDA error later. +2 regression tests; full scripts/tests 594 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): never advertise a GPU that CDI can't back, + honest GPU messaging (Bugbot) (#616) - Reconciler advertised GPU without CDI (HIGH): the boot reconciler re-asserted nvidia.com/gpu whenever /dev/dxg existed, even if `cdi generate` failed or left an incomplete spec. The installer already refuses in that case, but a Docker Desktop or Windows restart re-ran the reconciler and put capacity back onto a node where injection is broken -- pods schedule, then fail CUDA with no cluster-level signal. The reconciler now applies the SAME standard (cdi_ok: spec non-empty AND contains libdxcore AND `nvidia-ctk cdi list` parses) and simply doesn't start otherwise. - libdxcore path was fragile (MED): hardcoded /usr/lib/x86_64-linux-gnu/libdxcore.so, but it lives elsewhere on other Docker Desktop / WSL2 versions (/usr/lib/wsl/lib, the WSL driver store). A miss silently skipped the injection while the spec still passed the non-empty check -> GPU advertised, CUDA fails with the misleading driver error. Now probes the known locations, falls back to the linker cache, and mounts it at the path where it was actually found. - Green success before verification (MED): `Ok "GPU acceleration enabled (WSL2/CDI)"` printed at the capacity patch, before Confirm-GpuNode confirms allocatable GPU -- so a failed verification showed a green enabled line followed by a CPU fallback. Now an Info "verifying..." line; Confirm-GpuNode's "GPU verified and available" is the only success claim. - Misleading preflight warning + stale comments (LOW): the soft GPU preflight always said the node image "can't be built", including on the pull/mirror path where nothing is built locally -- now path-aware. Also corrected three comments that still claimed the device plugin is baked into the node image (this PR ships only the RuntimeClass). +4 tests; full scripts/tests 598 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): the in-WSL toolkit step no longer claims GPU is ready (Bugbot) (#616) Install-NvidiaContainerToolkit printed a green "GPU acceleration ready", set K3D_GPU_FLAG, and CLEARED GPU_SKIP_REASON after merely verifying the toolkit inside the WSL distro. But Confirm-DockerGpu is the authoritative gate and runs later, so: - operators could see a green ready line and then a CPU fallback, and - clearing the skip reason dropped the real cause when the later gate failed. It now reports only what it established ("NVIDIA Container Toolkit present in <distro>", Info not Ok) and logs that GPU is still gated on the Docker GPU probe. It no longer touches K3D_GPU_FLAG or GPU_SKIP_REASON. Nice invariant this establishes: K3D_GPU_FLAG is now assigned "--gpus=all" in exactly ONE place -- the authoritative gate -- which a new test pins, along with the toolkit step no longer setting the flag or clearing the reason. +1 test; full scripts/tests 599 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(gpu): machine-check the smoke-test JSON instead of eyeballing it (Bugbot) (#616) Bugbot reported an "extra closing brace" in the WSL2/CDI --overrides payload. Verified FALSE POSITIVE two ways: a JSON parser accepts it with the correct structure, and this is the exact escaped form that ran on a live box (kubectl created the pod; it failed later at CUDA, not at parsing). The `}}}` after "1" is correct -- it closes limits, then resources, then the container object. No behaviour change, but the concern deserves to be checkable rather than argued: extracted the command into a PURE Get-GpuSmokeTestCommand (selector in, string out), so tests can render both variants, strip the paste-escaping and ConvertFrom-Json them. A genuine brace slip now fails CI instead of shipping a command that errors on a healthy cluster. Also removes the duplication between the two branches and gives the diagnostics bundle one source of truth. +2 tests (JSON validity for the WSL2/CDI and device-plugin variants), and the previous source-scraping assertions now go through the builder. Full scripts/tests 601 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): close version-drift gaps that would cost GPU on a different Windows machine (#616) Audit prompted by "what breaks on someone else's laptop". The reassuring part first: a version mismatch CANNOT break the install. The k3d drop-in runs under `|| exit 1`, so a failure there would abort the node and fail cluster-create -- I verified under real `dash` (the node image's /bin/sh, no bashisms) that it exits 0 with the toolkit absent, with every nvidia-ctk subcommand failing, and with a garbage spec. It degrades to CPU instead. Two gaps found that would silently LOSE GPU on another machine: 1. The NVIDIA Container Toolkit was UNPINNED (`apt-get install nvidia-container-toolkit`), so two machines built weeks apart could get different builds. The whole WSL2 path leans on version-sensitive surfaces -- `cdi generate --mode=wsl`, `config --set nvidia-container-runtime.mode=cdi`, and the exact YAML the generator emits (our libdxcore injection parses it) -- so a future release changing the spec shape would break GPU on new installs while existing ones kept working. Pinned to the build validated on hardware (1.19.1-1) via ARG NCT_VERSION, and it FALLS BACK to latest if that version has aged out of the apt repo, so a stale pin degrades to "unpinned" rather than failing the build (which would cost GPU entirely). Also logs `nvidia-ctk --version` so the image records what it got. 2. The cdi_ok gate required `nvidia-ctk cdi list` to EXIST. That subcommand is version-dependent, so on a toolkit build without it we would have refused to advertise a perfectly working GPU -- a false negative on someone else's machine. The gate now uses format-stable structural facts (spec non-empty, declares nvidia.com/gpu, exposes /dev/dxg, carries the libdxcore mount) and uses `cdi list` only as an EXTRA veto when available. Verified both directions with a dash harness: a good spec + a toolkit lacking `cdi list` now advertises the GPU; a spec missing libdxcore still does not. +2 tests; full scripts/tests 603 pass / 0 fail; dash -n, check-style, check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gpu): every GPU-enable failure now names an actionable cause (#616) The residual risks that differ machine to machine all degraded to CPU correctly, but most collapsed into "the GPU node image build failed (docker build exit 1)" -- true, and useless to the person holding the laptop. Now each one names what happened and the single thing to do about it: - older Docker Desktop / no BuildKit labs frontend -> "update Docker Desktop, or point TRACEBLOC_K3S_CUDA_IMAGE at a prebuilt image" - full disk -> "free up space and re-run" - retired CUDA base tag -> names the tag + TRACEBLOC_CUDA_BASE_TAG - TLS-inspecting proxy (x509) -> TRACEBLOC_CA_BUNDLE - blocked/offline registry -> TRACEBLOC_IMAGE_REGISTRY (the air-gap path) - registry rate limit -> retry or use the mirror - anything else -> exit code + a pointer to the build output in the log Classification lives in a PURE Get-GpuBuildFailureReason so every branch is unit-tested. Also: - The Docker GPU probe failure now quotes the DETECTED driver version and a concrete minimum ("update to 525 or newer"). Our install gate accepts 460+, but WSL2 CUDA needs much newer, so "update your driver" left people unable to tell if theirs qualified. - GPU-detected-but-not-enabled gets its OWN summary block instead of being crammed into the Mode line: what happened, why, "fix that then re-run", and the log path. Previously the reason was a long parenthetical that was easy to miss. +12 tests; full scripts/tests 615 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): no green line may claim GPU is enabled before verification (Bugbot) (#616) Third report of this defect, so I swept for the whole family instead of patching one line. Fixed the reported one: the top-level gate printed Ok "GPU enabled -- cluster will use the custom k3s-CUDA image with --gpus=all" at Step 2, before cluster-create, the node's CDI wiring, and Confirm-GpuNode -- any of which can still clear K3D_GPU_FLAG. Now Info, worded as intent ("GPU support prepared … verified once the node is up"). The sweep found TWO MORE instances on the device-plugin path (pre-existing, not introduced here, but the same defect and the same misleading sequence): Ok "GPU acceleration enabled." both when the DaemonSet already existed and after a successful rollout -- printed before Confirm-GpuNode, which can find 0 allocatable GPUs and fall back to CPU. Both are now Info lines that say verification is pending. Message-only; no behaviour change on that path. Established the invariant instead of relying on review: Ok is reserved for facts already established, Info for intent. Exactly two Ok-level GPU lines remain, and both state something verified -- the image was built AND passed its k3s sanity check, and the node WAS observed advertising a GPU. A test pins the count at 2, asserts neither says "enabled", and asserts the four pre-verification lines are Info. +3 tests; full scripts/tests 618 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): the CDI revert must not require `cdi list` either (Bugbot) (#616) My own inconsistency, caught correctly. I made the cdi_ok gate tolerate a missing `nvidia-ctk cdi list` (it's version-dependent), but left the REVERT one block earlier calling it unconditionally. On a toolkit without that subcommand the sequence was: libdxcore injected -> `cdi list` "fails" (absent) -> edit reverted -> spec now lacks libdxcore -> installer reports "the spec is missing libdxcore" -> CPU i.e. a perfectly good injection thrown away, and a reason that is factually wrong (the library WAS found and mounted) and unactionable. The revert now reverts only when the parser is AVAILABLE and actively REJECTS the result -- the same rule as cdi_ok. Verified with a dash harness: with `cdi list` absent, libdxcore survives in the spec, the spec is still valid YAML, and the reconciler advertises the GPU. +1 test asserting every `cdi list` use is availability-gated: two probes, exactly two executable bare uses (comments excluded), and both guarded shapes pinned. Full scripts/tests 619 pass / 0 fail; dash -n, check-style, check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(gpu): shell-runnable drift guard for the embedded node-image files (review #633) Review asked for a check-facts/bats-style guard on the base64 embeds, having verified they are byte-identical today but noting nothing enforces it: check-facts guards the version PINS, not whole-file embedding, and Get-GpuBuildContentHash only hashes the EMBEDDED copies for local-build caching -- it never compares them to the repo files. A future edit to docker/k3s-cuda/* that isn't re-embedded would silently diverge (published image vs the installer's local build) with nothing failing. A Pester suite already asserts exactly this and runs in CI on windows-latest AND ubuntu-latest, so drift was in fact caught -- the reviewer simply had no pwsh to see it. That is a fair ergonomics gap, so this adds the bash-side mirror: decode each of the three blobs and diff against docker/k3s-cuda/*, plus non-empty checks (a truncated re-embed) and the drop-in shape invariant (no `exec /bin/k3s`, always `exit 0` -- the mistake that shipped once). CI already runs `bats scripts/tests/*.bats`, so it is picked up automatically. Verified the guard actually bites: appending a line to nvidia-runtimeclass.yaml without re-embedding turns test 2 red, and restoring it returns 5/5 green. On failure it prints the diff and the exact remedy (re-embed + regenerate the manifest). bats-hygiene passes (every assertion carries `|| return 1`, per #527). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): quote whitespace args + fail fast when GPU hosts are known-blocked (Bugbot) (#616) Two of four findings were real; the other two are refuted with evidence in the threads. 1. (Med) Arguments with whitespace could split. Invoke-BoundedProcess joins args into ONE command line, so an unquoted value containing a space -- a registry username, or a temp path under a profile like "C:\Users\First Last\..." -- silently became two arguments and corrupted the command. Now quotes any arg containing whitespace, leaves already-quoted values alone (so call sites that quote themselves aren't double-quoted), and preserves an empty string as a present-but-empty argument. Fixes the class, not just docker login. 2. (Med) A re-run on a restricted network looked hung. Preflight already soft-probes the GPU download hosts, but the gate then ran the probe (180s) and the build/pull (up to 15-20 min) anyway, so the "re-run to retry GPU" advice we print led to minutes of timeouts before the CPU fallback. Preflight now records the unreachable host and the gate short-circuits on it with an actionable reason (set TRACEBLOC_IMAGE_REGISTRY / a prebuilt image). Defaults empty, so a reachable machine is unaffected. +5 tests (incl. behavioural joiner cases: space-containing value, already-quoted path, empty string). Full scripts/tests 625 pass / 0 fail; embed-drift bats 5/5; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): only path-required hosts may skip GPU -- don't break air-gapped mirrors (Bugbot) (#616) A regression I introduced one commit earlier. The fast-fail short-circuit treated EVERY soft GPU host probe as blocking, but on a mirror / prebuilt-image install nvcr.io is unreachable BY DESIGN -- the node image and the CDI probe image are both re-homed to the mirror. So the "don't waste minutes timing out" optimisation disabled GPU for exactly the air-gapped case the mirror exists to serve, and the remedy told the operator to configure the mirror they had already configured. Probes now carry gpuBlocking, set only for the hosts the CURRENT path actually needs: - build path: nvcr.io + nvidia.github.io -> blocking - mirror/prebuilt path: the configured GPU registry host -> blocking; nvcr.io still probed (a warning is useful) but NON-blocking Only a blocking probe arms GPU_HOSTS_UNREACHABLE. The skip remedy is now path-aware too: on the mirror path it says to check that the configured GPU registry is reachable and holds the k3s-CUDA image, rather than suggesting a mirror. +2 tests pinning both directions. Full scripts/tests 627 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The foundational build artifact for GPU-enabled Windows edges (#616), split from the installer PR (#633) so it can land on
developand be published via CI — GitHub only dispatches aworkflow_dispatchworkflow from the default branch.Inert on develop: nothing references these files until #633 wires the installer to the image, so this is a safe, behavior-free landing.
Why a custom image
The stock
rancher/k3simage is Alpine with no NVIDIA container runtime, so GPU pods can never schedule on it (the node advertises0 nvidia.com/gpu). This image rebuilds the same pinned k3s (K3S_TAG== installerK8S_VERSION) on an NVIDIA CUDA Ubuntu base, installs the NVIDIA Container Toolkit, configures containerd for thenvidiaruntime, and bakes in the device plugin +nvidiaRuntimeClass so the node advertisesnvidia.com/gpuon first boot. Based on the official k3d CUDA recipe.Contents
docker/k3s-cuda/Dockerfile— multi-stage k3s + CUDA basedocker/k3s-cuda/nvidia-device-plugin-daemonset.yaml— device plugin +nvidiaRuntimeClass (v0.14.5)docker/k3s-cuda/build.sh,README.md.github/workflows/build-k3s-cuda.yaml— manual dispatch;push: truepublishes to GHCR and validates the Dockerfile buildsAfter merge
Run
build-k3s-cudawithpush: trueto publishghcr.io/tracebloc/k3s-cuda:<k3s>-cuda-<base>(make the package public so it pulls with no auth). Then the installer (#633) pulls it automatically at cluster-create — the end-user runs one command, no manual build or pull, GPU or CPU auto-detected.Part of #616.
🤖 Generated with Claude Code
Note
Low Risk
New build assets and CI drift gates only; no installer or runtime behavior changes until a follow-up PR wires the image.
Overview
Introduces
docker/k3s-cuda/— a multi-stage image that layers the same pinned k3s as CPU nodes onto an NVIDIA CUDA Ubuntu base, installs the Container Toolkit, configures containerd’snvidiaruntime, and auto-applies a baked device plugin +nvidiaRuntimeClass (v0.14.5) on first boot.build.shtags images as<K3S_TAG>-cuda-<CUDA_TAG>for deterministic pulls once the installer is wired (#633).Adds a manual
build-k3s-cudaworkflow (workflow_dispatch, optional GHCR push, 60m timeout, unpinned docker/buildx only) so the large CUDA build does not run on every push.Extends
check-facts.sh(and installer-tests path filters + bats fixtures) soK3S_TAGin the Dockerfile,build.sh, and the workflow’sk3s_tagdefault must matchfacts.envK8S_VERSION, preventing a k8s bump from referencing an unpublished GPU image.Inert until #633: nothing in this PR switches cluster create to the new image; it only lands the artifact and CI guardrails.
Reviewed by Cursor Bugbot for commit 76b2eaf. Bugbot is set up for automated code reviews on this repo. Configure here.