fix(installer): harden the Windows install→first-experiment path — checksum-driven downloads, reliable cluster-create, writable ingest & training volumes, local-chart support (#611) by shujaatTracebloc · Pull Request #612 · tracebloc/client

shujaatTracebloc · 2026-08-05T14:19:42Z

What & why

Hardens the Windows install → first-dataset-ingest → first-experiment path end to end, driven by real field failures on a Windows machine behind a filtering corporate/AV network. Follow-on to the resilient multi-transport download landed in #607/#608. Each fix below is a place a real user got stuck; together they take the flow from "dies at step 2" to "installs, provisions, ingests, and runs an experiment" on a hostPath (Windows/WSL2) install.

Closes#611. Contributes to #578.

Fixes

1. Checksum-driven tool download (k3d / kubectl / helm) — `install-k8s.ps1`

#607's multi-transport download validated by size + magic bytes only, so a binary truncated mid-transfer to between the size floor and the real size still passed and dead-ended at a no-retry checksum step. Now the checksum is the completeness test:

Get-VerifiedDownload -Sha256 — after a transport lands a size/magic-valid file, its SHA-256 must match or the next transport (curl.exe → BITS) is tried. A truncated/altered copy self-heals.
The checksum-list files (checksums.txt, *.sha256) are fetched first and gated on a hash-anchored-MatchPattern (a 64-hex hash adjacent to the asset / at the start of the body), so a proxy error page echoing the URL can't fail the gate open (Bugbot).
helm gains checksum verification on the PS path for the first time (parity with bash).
Field symptom fixed: System tool checksum verification failed on a correct-but-truncated k3d download.

2. Reliable cluster-create — `install-k8s.ps1`

Wait-ProcessWithDeadline polled HasExited but never called WaitForExit(), so with Start-Process -RedirectStandardOutput the k3d process's ExitCode came back null and $null -ne 0 misread a successful create as a failure. Now it calls WaitForExit() before returning (reliable exit code for every caller), plus a defense-in-depth fallback that trusts k3d's created successfully marker — checking both stdout and stderr (k3d logs success via logrus to stderr) (Bugbot).

Field symptom fixed: Failed to create compute environment even though the cluster was up.

3. Writable hostPath volumes for ingest and training/inference — chart (`jobs-manager-deployment.yaml`)

Two permission failures, same root cause. On hostPath installs kubelet doesn't apply fsGroup (kubernetes/kubernetes#138411), so a hostPath dir is created root-owned and the non-root pods that mount it can't write:

Ingest:tb data ingest failed at the copy step — mkdir: can't create directory '/data/shared/.tracebloc-staging/': Permission denied.
Training / inference: a spawned pod failed with PermissionError [Errno 13]: '/data/logs/<run>' while creating its per-run log dir (os.makedirs).

mysql-data already had a privileged init-chown; the shared and logs volumes had none. Now, on hostPath only, jobs-manager runs a privileged init-writable-data (root, caps CHOWN+FOWNER+FSETID) that makes both /data/shared and /data/logschmod 3777 — world-writable (multiple non-root writers whose UIDs the chart doesn't control: jobs-manager, the spawned training/inference pods, the CLI ingest-staging pod) + setgid (GID-1000 inheritance; FSETID is required or the kernel strips setgid after the chown) + sticky (one writer can't unlink another's files). The spawned pods mount the same hostPath dirs, so fixing them once reaches them too. Training and inference share one pod spec (client-runtime job.yaml), so /data/logs covers both; every other write path there is an emptyDir or read-only. Each dir is fixed independently and best-effort, so a chown that can't complete (NFS root_squash) doesn't wedge the edge. No fsGroup is set — it's a no-op on hostPath and would strip the spawned pods' group-0 access on CSI; CSI is untouched (matches develop) and its pods keep their documented posture (docs/SECURITY.md §5.3).

4. Windows local-chart support (`TRACEBLOC_CHART_PATH`) — `install-k8s.ps1`

The Windows installer could only install the published chart, so branch-only chart fixes (like #3 above, or #585's global.imageRegistry) were untestable from a Windows install. Added TRACEBLOC_CHART_PATH — parity with the bash installer's _resolve_chart_ref — so a local chart directory can be installed (skipping helm repo add); the published repo remains the default.

Tests

Pester source-guards + unit tests for every installer item above (checksum-driven retry, hash-anchored gates, WaitForExit, both-stream fallback, local-chart ref). Full suite green (451).
helm-unittest for the writable-volume init: present on hostPath / absent on CSI, no fsGroup on either path, and asserts the init has caps CHOWN+FOWNER+FSETID, runs the per-dir best-effort loop with chmod 3777, and mounts both/data/shared and /data/logs (mounting both is required or the chown is a no-op). Chart bumped 1.9.15 → 1.9.20 across the chart-affecting fixes (chart-content guard).
Windows-facing scripts are manifested; manifest.sha256 regenerated. bats/macOS suites unaffected (Linux/macOS paths untouched except the already-merged Installer tool acquisition must survive a proxy/AV-truncated binary download (k3d/kubectl/helm) #607 shared-bash bits, which a full cross-OS audit confirmed safe).

Scope note

This PR spans the installer scripts and the chart because they're the two halves of the same Windows install→ingest→experiment flow being validated together (kept in one PR by request).

The chart change is confined to the hostPath path: the privileged init-writable-data container is gated to hostPath and sets no fsGroup, so managed (EKS/AKS/OC) installs are genuinely untouched — CSI matches develop, and its spawned pods keep their documented posture (docs/SECURITY.md §5.3). (An earlier revision set fsGroup: 1000 on jobs-manager; @saadqbal and Bugbot correctly flagged that it never reaches the spawned writers and its relabel would strip their group-0 access on CSI, so it was removed.)

CI note

If pii-gate / pii-check is red, it is failing closed because the org PII_DENYLIST secret is unavailable — an org-wide infra issue affecting every PR, not this PR's content (tracked separately; the pii-gate-override label doesn't currently bypass it — tracebloc/.github#165). All other checks and Cursor Bugbot are green.

🤖 Generated with Claude Code

…ated tool binary self-heals (#611) Field follow-up to #607/#608. On a Windows machine behind a filtering proxy, the k3d download kept failing at "System tool checksum verification failed" even with #608's multi-transport download — because #608 validated a download by SIZE FLOOR + magic bytes only. A binary truncated mid-transfer to somewhere between the 10 MB floor and the real 25.8 MB still passed (it's >10 MB and starts with 'MZ'), so the fallbacks never fired and it dead-ended at the separate, no-retry checksum step. (Proven on the box: a manual download produced the correct hash at 25,805,312 bytes, while the installer's copy failed the checksum in ~2s with no fallback.) Fix — the checksum is the authoritative completeness test: - Get-VerifiedDownload gains -Sha256: after a transport lands a size/magic-valid file, its SHA-256 must equal the expected hash or the transport is treated as failed and the NEXT one (curl.exe -> BITS) is tried. A truncated/altered copy now self-heals instead of dead-ending. - Get-VerifiedDownload gains -MustContain for the small checksum-list files, so a proxy error page lacking the expected asset line is retried too. - k3d / kubectl / helm now fetch their checksum FIRST (resiliently) and pass the extracted, 64-hex-validated hash as the download gate. helm gains checksum verification on the PS path for the first time (parity with the bash path). Tests: Pester source-guards for the -Sha256/-MustContain gates, the mismatch->retry path, and per-tool checksum-first wiring; full suite green (444). Manifest regenerated. Contributes to #578. Closes#611. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ries transports (Bugbot) The kubectl .sha256 is a bare 64-hex hash with no fixed substring, so it used -MinBytes 1 with no content gate -- a proxy error page satisfied the floor, the first transport 'succeeded', curl.exe/BITS never ran, and the later hex check aborted. Add -MatchPattern (a regex content gate) to Get-VerifiedDownload and use '[0-9a-fA-F]{64}' for the kubectl checksum fetch, matching how k3d/helm use -MustContain. Pester guards updated; manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ed (#611) Field report (same Windows box, past the k3d download fix): Step 3 aborted with "Failed to create compute environment" even though k3d printed "Cluster 'tracebloc' created successfully!" with EMPTY stderr and the cluster was actually up. Cause: Wait-ProcessWithDeadline polled HasExited but never called WaitForExit(), and Start-Process -RedirectStandardOutput can leave $proc.ExitCode $null in that window -- so `$null -ne 0` misread an exit-0 success as a failure. Not machine-specific; a latent race any Windows user can hit. - Wait-ProcessWithDeadline now calls $Process.WaitForExit() before returning success, so the redirected streams drain and ExitCode is reliable for EVERY caller (cluster create, partial delete, tracked installs). - Cluster-create adds defense-in-depth: a still-null exit code falls back to k3d's own "created successfully" marker rather than failing a cluster that is up. - Pester source-guards for both. Windows-only change (install-k8s.ps1); Linux/mac paths untouched and their suites remain green. Contributes to #578. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ms in the exit fallback (Bugbot) Two Bugbot findings on the checksum-driven download work: 1. (High) The checksum-LIST fetch gates were fail-open, so a proxy error page "succeeded" on the first transport and skipped the curl.exe/BITS retry — exactly the case #611 exists to survive. Helm's -MustContain substring (helm-<ver>-windows-<arch>.zip) also appears in the request URL a proxy page can echo; kubectl's -MatchPattern was unanchored so any page with a 64-hex run passed; k3d gated on the bare asset name. Fix: drop the weak -MustContain entirely and gate every checksum-list fetch on the hash STRUCTURE — k3d/helm require a 64-hex hash adjacent to the asset, kubectl requires the hash at the start of the body. A proxy/HTML error page can't satisfy that, so it retries transports as intended. 2. (Medium) The null-exit-code cluster-create fallback only scanned $k3dStdout, but k3d logs its "Cluster created successfully!" line via logrus to STDERR — so a real success could be misread as failure. Fix: check both $k3dStdout and $k3dStderr. Pester source-guards updated: kubectl gate is start-anchored, k3d/helm gates are hash-anchored, no -MustContain remains, and the fallback inspects both streams. Full suite green (447). Manifest regenerated. Contributes to #578. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…Path installs `tb data ingest` failed at the copy step with `mkdir: can't create directory '/data/shared/.tracebloc-staging/': Permission denied`. On hostPath installs (the Windows/WSL2 + bare-metal default) kubelet does not apply fsGroup to hostPath volumes (kubernetes/kubernetes#138411), so /data/shared (client-pvc) is created root-owned and the non-root ingest-staging pod can't write to it. mysql-data has a privileged init-chown for exactly this reason; the shared data volume had none. - jobs-manager gains fsGroup: 1000 (CSI clusters apply it to the shared volume). - On hostPath, a privileged init-shared-data container (root, CHOWN+FOWNER only) chowns /data/shared to 1000:1000 and chmod 2777. World-writable, unlike mysql-data's single-UID chown, because the shared volume has multiple non-root writers whose UIDs this chart doesn't control -- jobs-manager, the training/ ingestor pods it spawns, and the CLI's ingest-staging pod. setgid keeps new files in GID 1000; the init is gated on hostPath (CSI relies on fsGroup). - helm-unittest: init present + world-writable on hostPath; absent (fsGroup kept) on CSI. Chart bumped 1.9.15 -> 1.9.16. Client-side companion to the installer fixes on this PR (requested to land here). Contributes to #578. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… parity with bash) The Windows installer could only ever install the PUBLISHED chart (helm repo), so a branch-only chart change (e.g. the #611 /data/shared fix, or #585's global.imageRegistry) was impossible to test from a Windows install. The bash installer already supports a local chart via TRACEBLOC_CHART_PATH (_resolve_chart_ref); this brings Windows to parity. - When $env:TRACEBLOC_CHART_PATH is set, install-k8s.ps1 installs from that local chart directory (validated) and skips `helm repo add`; otherwise it uses the published repo as before. Applied to both the fresh-install and adopt/reconcile helm upgrades. - Pester source-guards for the local-chart ref, the repo-add skip, and the not-a-directory error. Manifest regenerated. Full suite green (451). Enables a from-scratch Windows test of the branch chart. Contributes to #578. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

saadqbal

Careful, well-tested PR — the checksum-as-completeness-test and the both-streams k3d exit fallback are the right calls. A few non-blocking things to weigh, mostly on the chart side (left inline).

#611) hostPath ignores fsGroup (kubernetes/kubernetes#138411), so /data/logs was created root-owned and non-root training/inference pods hit `PermissionError [Errno 13]` creating their per-run log dir (`os.makedirs('/data/logs/<run>')`). The #611 init-container chowned /data/shared but not /data/logs — the same class of bug on the logs volume. Extend the init (renamed init-shared-data -> init-writable-data) to chown+chmod BOTH hostPath volumes and mount both. Training and inference pods share one spec (job.yaml), so this covers both; ingestion already covered by the /data/shared chmod. Bump chart 1.9.16 -> 1.9.17. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…riven-download

…en-reuse parity Two reviewer follow-ups on #612: - chart: chmod the writable hostPath dirs 3777 (was 2777) — add the sticky bit so one writer can't unlink/rename another writer's files in /data/shared // /data/logs (/tmp semantics). setgid is retained. Safe given the uid topology (dir owned by 1000; training pods run as 1000; the ingestor writes its own subtrees as a stable uid) and no cross-uid filesystem deletes exist in client-runtime. Chart 1.9.17 -> 1.9.18. - install-k8s.ps1: the adopt/reconcile helm upgrade now prefers --reset-then-reuse-values when `helm upgrade --help` advertises it (Helm >= 3.14), falling back to --reuse-values otherwise — so NEW chart defaults reach adopted Windows edges on auto-upgrade (bash parity with install-client-helm.sh). manifest.sha256 regenerated. Tests: helm-unittest updated (3777); new Pester test asserts the reset-then-reuse preference; full Pester suite 452 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…bot) init-writable-data chowns the shared/logs hostPath dirs to GID 1000 then chmods 3777. With caps dropped to CHOWN+FOWNER only, the kernel silently strips S_ISGID on the chmod — after the chown the dir's group no longer matches the process (fsgid 0), and a root process without CAP_FSETID can't set setgid on it — so the mount landed at 1777 and new files did NOT inherit GID 1000 as documented. Add FSETID to the cap set; setgid now sticks. helm-unittest asserts FSETID present. Chart 1.9.18 -> 1.9.19. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 15e0f37. Configure here.}

… (Bugbot) Two Bugbot findings on the writable-volume fix: - HIGH — remove `fsGroup: 1000` / `fsGroupChangePolicy` from jobs-manager. It is a no-op on hostPath (kubelet ignores fsGroup — the init does the work) and on CSI it only grants jobs-manager's OWN processes GID 1000 while its OnRootMismatch relabel flips the shared/logs volumes to group 1000 — stripping the group-0 access the spawned training pods (UID 1001 / OpenShift arbitrary UID, GID 0) and the host-UID ingestion pods rely on (docs/SECURITY.md §5.3). It never reaches those spawned writers, so it was all regression risk and no gain. Those pods keep their own documented posture; CSI is untouched (matches develop). - MEDIUM — the init now fixes each dir INDEPENDENTLY and best-effort: `for d in /data/shared /data/logs; do chown && chmod || echo <warn>; done`. A chown that can't complete (e.g. /data/shared on an NFS root_squash export) no longer aborts the chain and skips /data/logs — the other dir is still repaired and jobs-manager still starts; a truly unwritable mount surfaces as a clear error at the writer pod instead of wedging the edge in Init. helm-unittest updated (no fsGroup on either path; per-dir loop; CSI skips init). Chart 1.9.19 -> 1.9.20. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

After the rebase merged our port-6550 Pester tests with #612, Test-Preflight's network-reachability block calls Has for kubectl/helm/k3d. The "port 6550 in use by our running cluster -> ok" case passes ownership, so it now continues into that block, where a k3d-only -ParameterFilter left those later Has calls with no matching mock ("No mock for command 'Has' matched"). Use a plain default mock (k3d + tools present -> only always-critical hosts probed) so every Has call is covered and the case does not throw. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…#622) * fix(#557): preflight port 6550 on Windows so a conflict fails clearly New-K3dCluster binds the cluster API server to 127.0.0.1:6550, but Test-Preflight never checked whether 6550 was already in use. On a machine where a leftover/other k3d cluster or an unrelated service owns 6550, `k3d cluster create` failed and the installer surfaced k3d's raw stderr instead of a clear "port already in use" cause. Add Get-PfPortListening (Get-NetTCPConnection based; $null when undeterminable) and a port-6550 check to Test-Preflight. A port owned by THIS installer's own already-running cluster is fine (that run reuses it), so the check only hard-fails when the listener is NOT our cluster, with actionable guidance to find and stop the owner. Adds Pester coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#557): harden port-6550 preflight (bounded, fail-closed, running-gated) Address PR #622 review on the Windows port-6550 preflight: - Get-PfPortListening no longer fails open: -ErrorAction SilentlyContinue swallowed real CIM/access errors into the same empty result as a free port, so a busy port green-OK'd. Switch to -ErrorAction Stop and, in the catch, treat only Get-NetTCPConnection's ObjectNotFound (no listener) as "free"; every other error returns $null ("can't tell"). (Bugbot Med) - Port-ownership check now reuses Test-ClusterRunning instead of a bare `k3d cluster list -o json`: it wraps the call in the same ~15s job deadline, so a wedged Docker engine can't hang preflight (Bugbot High), and it gates on serversRunning >= 1, so a STOPPED leftover cluster no longer masks a foreign listener on 6550 (Bugbot Med). - Regenerate scripts/manifest.sha256 for the changed install-k8s.ps1. - Add Pester coverage: probe error -> $null, ObjectNotFound -> free, stopped-leftover + foreign listener -> fail, running-owned -> ok. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(#557): give the port-6550 running-cluster case a default Has mock After the rebase merged our port-6550 Pester tests with #612, Test-Preflight's network-reachability block calls Has for kubectl/helm/k3d. The "port 6550 in use by our running cluster -> ok" case passes ownership, so it now continues into that block, where a k3d-only -ParameterFilter left those later Has calls with no matching mock ("No mock for command 'Has' matched"). Use a plain default mock (k3d + tools present -> only always-critical hosts probed) so every Has call is covered and the case does not throw. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#557): tri-state port-6550 ownership so a slow k3d list can't hard-fail reuse (Bugbot 3728340365) The port-6550 ownership check decided ownership solely from Test-ClusterRunning, whose bounded `k3d cluster list` maps BOTH a timeout AND a parse failure to $false. The ownership block treated that $false as "foreign listener" and HARD-FAILED preflight with stop/delete hints, so a slow/wedged Docker on a normal re-run wrongly blocked the install and pointed the operator at their own cluster -- conflating "can't determine" with "definitely not ours". Make cluster run-state tri-state: - Get-ClusterRunStateFromList (pure) and Get-ClusterRunState (bounded) return 'running' | 'down' | 'unknown'. Get-ClusterRunState reports 'unknown' only when the list times out or its output is unparseable. - Test-ClusterRunningInList / Test-ClusterRunning become thin boolean wrappers (=='running'), preserving their existing contracts and callers. The ownership block now only hard-fails when CONFIDENT the listener is foreign ('down': enumerated and ours is absent/stopped, or no k3d installed). An 'unknown' read downgrades to a warning and proceeds, letting New-K3dCluster's start/repair path settle it instead of blocking the install. Tests: existing 3 cases updated to mock Get-ClusterRunState; added an indeterminate case (list timed out -> warns, does NOT hard-fail). Manifest regenerated. Pester not run locally (pwsh unavailable); validated by source analysis. Source-of-truth guards for Test-ClusterRunning still hold. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#557): classify an absent cluster as 'down', not 'unknown' (Bugbot High 3728714531) Get-ClusterRunState ran `k3d cluster list <name> -o json`, which fatals with empty stdout when the named cluster is absent. The classifier read that empty output as 'unknown' -> warn-and-proceed, so a genuinely-absent cluster with a FOREIGN listener on 6550 escaped the hard-fail -- exactly the "re-run after the first create failed on the busy port" case #557's preflight must catch. List ALL clusters instead (`k3d cluster list -o json`, no name) and look for $CLUSTER_NAME in the parsed array. A successful full list always emits at least `[]`, so absent-vs-error stays separable: - completed list without a running $CLUSTER_NAME (absent, stopped, or empty []) -> 'down' (confidently not ours -> hard-fail on a busy port) - only a TIMED-OUT or unparseable/failed list -> 'unknown' (warn-and-proceed) Tests: added a Get-ClusterRunState tri-state Describe -- pure classifier cases (absent/empty-list -> 'down'; empty/garbage -> 'unknown') plus bounded cases (timeout -> 'unknown'; completed-but-absent -> 'down'). Manifest regenerated. Pester not run locally (pwsh unavailable); validated by source analysis. Source-of-truth guards for Test-ClusterRunning still hold. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(#557): drop fragile job-cmdlet-mock tests for bounded Get-ClusterRunState (CI Pester) The two bounded-wrapper cases mocked Start-Job/Wait-JobWithProgress/Receive-Job, which is environment-fragile and fails under CI Pester. Coverage is retained via the pure Get-ClusterRunStateFromList tests + the Test-ClusterRunning source-guard (bounded-job regex). No product-code change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Resolve conflicts after today's develop advances (#612 chart bumps → 1.9.21, #635 build-infra + check-facts k3s-cuda consumers + installer-tests paths): - scripts/manifest.sha256: the only content conflict — regenerated against the merged tree (authoritative for the final install-k8s.ps1). - docker/k3s-cuda/* and build-k3s-cuda.yaml: took develop's versions (they live on develop now via #635; #633 had removed its copies to avoid an add/add). Verified on the merged tree: install-k8s.ps1 parses clean (GPU funcs intact), check-facts --check passes, helm-unittest 32/32, full Pester 480 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…d no login (#616) Restores the one-command promise for GPU. The GPU node image was published private, so enabling GPU forced the user to supply a registry token -- exactly the friction #612 (CPU) never had, because every image it pulls is public. Default GPU path is now a LOCAL build: on an NVIDIA machine the installer runs `docker build` from PUBLIC bases only (rancher/k3s + NVIDIA's public nvcr.io CUDA base + the public NVIDIA container toolkit), tags it locally, and creates the cluster with that image -- no registry login, no private package, one command. Build-GpuNodeImage is idempotent (reuses an already-built image), bounded with a visible progress bar (Wait-ProcessWithDeadline), sanity-checks that the built image runs k3s, and on any failure falls back to CPU with a clear reason. The Dockerfile + device-plugin manifest are embedded (self-contained verified artifact) and a Pester drift guard keeps them byte-identical to docker/k3s-cuda/*. An explicit prebuilt image / mirror (TRACEBLOC_K3S_CUDA_IMAGE or TRACEBLOC_IMAGE_REGISTRY, i.e. air-gap tenants) still PULLS instead of building. CPU is untouched: the whole block is behind the NVIDIA gate + the docker-run probe, so a CPU machine builds nothing and gets stock rancher/k3s exactly as before. Windows-installer only; staging/prod (Linux + chart) are not affected. Full Pester 499 pass; check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…rt (#616) (#633) * fix(installer): Layer 1 — never request a GPU the cluster can't provide (#616) On Windows a machine WITH a working NVIDIA GPU still ran CPU-only: the installer requested nvidia.com/gpu for every job whenever it merely DETECTED a GPU + driver, but the GPU is only provisioned in the cluster when K3D_GPU_FLAG is set (the WSL NVIDIA-toolkit step fully succeeded). When that step didn't complete, jobs got a GPU request the node couldn't satisfy -> Pending "Insufficient nvidia.com/gpu" until the SINGLE_NODE fallback downgraded them to CPU. GPU wasted, silently. Layer 1 (safe, always-correct; CPU fallback preserved): - Gate GPU_REQUESTS/GPU_LIMITS on K3D_GPU_FLAG (the SAME condition that provisions the GPU), not on mere detection. Empty gpuVal => no GPU request => training runs on CPU. The SINGLE_NODE downgrade stays as a belt-and-suspenders net. - Make the skip LOUD: capture a specific $GPU_SKIP_REASON at each toolkit early-return (WSL not responding / Ubuntu needs first-run setup / toolkit apt blocked or timed out / verify failed), surface it in the install summary ("CPU (GPU detected but not enabled: <reason>)") and warn during install. - Gate the doctor's GPU-test hint on K3D_GPU_FLAG too (don't suggest a test that can't run). Layer 2 (actually enabling the GPU in k3d-on-Windows) follows in this PR. Tests: 4 new Pester tests (values carry no GPU request when not enabled / carry it when enabled; summary shows CPU+reason vs NVIDIA GPU). Full suite 456 pass. manifest.sha256 regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): Layer 2a — custom k3s-CUDA node image for GPU edges (#616) The stock rancher/k3s image is Alpine-based with no NVIDIA container runtime, so GPU pods can never schedule on it. Add a drop-in replacement k3s node image that rebuilds the SAME pinned k3s (K3S_TAG == installer K8S_VERSION) on an NVIDIA CUDA Ubuntu base, installs the NVIDIA Container Toolkit, configures containerd for the `nvidia` runtime, and bakes in the device plugin + `nvidia` RuntimeClass so the node advertises nvidia.com/gpu on first boot. Based on the official k3d CUDA recipe. - docker/k3s-cuda/Dockerfile (multi-stage k3s + CUDA base) - docker/k3s-cuda/nvidia-device-plugin-daemonset.yaml (device plugin + RuntimeClass, pinned v0.14.5) - docker/k3s-cuda/build.sh (build/push, tag encodes k3s + CUDA versions) - docker/k3s-cuda/README.md - .github/workflows/build-k3s-cuda.yaml (manual dispatch; push=true publishes to GHCR) Image tag encodes both the k3s pin and CUDA base so a new k8s pin can't silently reuse a stale GPU image. The installer wiring to actually USE this image (cluster --image swap, GPU capability probe, RUNTIME_CLASS_NAME=nvidia) lands next in this PR; CPU fallback (Layer 1) always remains. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): Layer 2b — wire the installer to the custom k3s-CUDA image (#616) Make a Windows NVIDIA GPU genuinely usable by training pods, with CPU fallback whenever it can't be wired up. - Authoritative GPU gate: add Confirm-DockerGpu, which runs `docker run --rm --gpus all nvidia/cuda:<tag> nvidia-smi`. Docker Desktop uses its OWN WSL distro, so the toolkit-in-Ubuntu step was never a reliable signal; actually running a GPU container is. Enable GPU iff the probe passes -> we never create a `--gpus` cluster that would fail, and DD-GPU works even without the user's Ubuntu toolkit. Probe failure => K3D_GPU_FLAG cleared + a clear reason => CPU (Layer 1). - Cluster create: use the custom k3s-CUDA image ($K3S_CUDA_IMAGE, env-overridable via TRACEBLOC_K3S_CUDA_IMAGE, default ghcr.io/tracebloc/k3s-cuda:<K8S_VERSION>-cuda-<base>) when GPU is enabled; stock rancher/k3s otherwise. Same pinned k3s either way. - Set RUNTIME_CLASS_NAME=nvidia in values when GPU is enabled, so jobs-manager threads runtimeClassName: nvidia into every spawned pod (the RuntimeClass is baked into the CUDA image). Empty otherwise. Tests: Confirm-DockerGpu behavioral tests (pass/fail/no-GPU) + source guards for the image swap, probe gating, and RuntimeClass; existing GPU values tests extended for RUNTIME_CLASS_NAME. Full Pester suite 464 pass. manifest.sha256 regenerated. NOTE: end-to-end GPU execution needs validation on a real GPU Windows box (build + publish the image via the build-k3s-cuda workflow, then re-install). CPU path is unaffected and fully covered. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): make the GPU node image mirror-aware (one-command, air-gap) (#616) The one installer command must do everything with no external steps — the user never builds or pulls the GPU image by hand; the installer pulls it automatically at cluster-create. Re-home the default k3s-CUDA image onto TRACEBLOC_IMAGE_REGISTRY when a private mirror is configured (#585), so the single command also works on a restricted/air-gapped network, same as every other image. Explicit TRACEBLOC_K3S_CUDA_IMAGE still overrides. Source guard updated. Pester 464 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(installer): move k3s-CUDA image build infra to its own PR (#635) The Dockerfile + build workflow now live in #635 (targeting develop) so the image can be published via CI before this installer PR merges. #633 keeps only the installer wiring that CONSUMES the published image; nothing here changes. After #635 merges + the image is published, this branch rebases on develop to pick the build infra back up transitively. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): pull the private GPU node image with creds — keep it private (#616) We keep ghcr.io/tracebloc/k3s-cuda PRIVATE rather than public. So the installer must authenticate to pull it — the end user still runs ONE command; the creds come from env, not a separate docker login. - Confirm-GpuImagePullable: derives the registry host from $K3S_CUDA_IMAGE, `docker login`s with TRACEBLOC_REGISTRY_USERNAME/PASSWORD (the same vars the mirror uses, #585) when set, then `docker pull`s the image — which both VERIFIES access and PRE-LOADS it so k3d cluster-create reuses the local copy (no second pull). - The GPU gate now requires BOTH the passthrough probe AND a pullable image: `if ((Confirm-DockerGpu) -and (Confirm-GpuImagePullable))`. Either failure => clean CPU fallback with a specific reason (private-registry hint names the exact env vars to set; cred-set-but-failed says to check read access) instead of a cluster-create that dies pulling an unauthorized image. Tests: 4 Confirm-GpuImagePullable cases (login+pull ok / cred fail / no-cred hint / gate source-guard); updated the stale probe-gate guard. Full Pester 468 pass. manifest regen. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): bound the GPU docker calls with a timeout (Bugbot) (#616) The new `docker run` (GPU probe), `docker login`, and `docker pull` calls were unbounded — a wedged Docker daemon, registry, or proxy could hang Step 2 forever instead of falling back to CPU (installer external-call timeout rule). Route all three through a new Invoke-DockerCli helper that runs docker in a bounded background job (Wait-JobWithProgress deadline); on timeout it kills the job and returns Code=124 so callers fall back to CPU cleanly. Timeouts: probe 180s, login 60s, pull 900s (the GB CUDA image on a slow/proxied network — generous but finite). Login stdin (the token) is passed in-memory via the arg hashtable, never to disk, argv, or logs. Tests: rewired Confirm-DockerGpu / Confirm-GpuImagePullable tests to mock Invoke-DockerCli (regular function — mockable, unlike a job'd docker), added timeout-path (Code 124 -> CPU) cases + a source guard that every GPU docker call is bounded. Full Pester 482 pass. manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): drift-guard the CUDA tag via facts.env (Bugbot) (#616) CUDA_BASE_TAG (the CUDA base that derives the pulled GPU image tag) was hardcoded in four places — install-k8s.ps1, docker/k3s-cuda/Dockerfile ARG, build.sh, and the build-k3s-cuda workflow default — outside facts.env / check-facts, so a CUDA bump in one spot could leave the installer pulling a GPU image tag that was never built (#547-class drift, same as the k3s pin). Add CUDA_TAG=12.4.1-base-ubuntu22.04 to facts.env and wire all four consumers into check-facts.sh (verified == facts.env; --write restamps them). The workflow extractor distinguishes the cuda_tag default (…-ubuntu…) from the k3s_tag default (vX…k3s). check-facts.bats fixture seeds the two new consumers. check-facts.sh --check passes (15 consumers), check-facts.bats 14/14, shellcheck clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): kill docker on timeout — no orphaned native process (Bugbot) (#616) Invoke-DockerCli used Start-Job; Stop-Job stops the PS job but can orphan the native docker.exe it spawned, so a timed-out run/login/pull could keep going after the installer fell back to CPU. Run docker as a direct child Process instead and Kill() it on timeout, so the deadline actually terminates docker. 5.1-safe (.NET Framework Process API); args have no spaces (flags + image tags) so a plain join is safe; login stdin is written in-memory, never to disk/argv/logs. Callers are unchanged (still mock Invoke-DockerCli); updated the source guard to assert WaitForExit(timeout) + Kill(). Full Pester 482 pass. manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): accurate GPU-skip reasons + drift-check the GPU image (Bugbot) (#633) Two Bugbot findings: - GPU_SKIP_REASON now reflects the ACTUAL failure. A probe/pull timeout (Code 124) reports "timed out", not a GPU-unavailable or credentials error; a real GPU passthrough failure reports the Docker Desktop hint, not stale Ubuntu guidance. - Test-K3sVersionDrift now recognizes the GPU node image (…/k3s-cuda:<k3s>-cuda-<base>), extracting the k3s pin from before "-cuda-", so a GPU cluster no longer silently escapes the version-drift check after a pin bump (it only parsed rancher/k3s: before). Tests: reason assertions on the timeout/non-zero paths + a drift source guard for the GPU image. Full Pester 483 pass. manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): single-node GPU cluster so one card isn't double-counted (Bugbot) (#616) k3d's --gpus=all exposes the SAME host GPU to every node container, and the baked-in device-plugin DaemonSet registers it once per node. A default server+agent cluster therefore advertises nvidia.com/gpu=1 on BOTH nodes -- 2 allocatable for 1 physical card -- and Kubernetes can schedule two jobs onto the same device. Extra k3d nodes live on the same Docker host and all see the same card, so multi-node can never add real GPUs; it only double-counts. When the GPU is enabled, collapse to a single node (agents=0) so the card is advertised exactly once. An explicit user AGENTS is overridden loudly, not silently. CPU installs are unchanged (topology only shifts inside the K3D_GPU_FLAG="--gpus=all" branch). +1 source-guard test; full Pester 484 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): reused CPU cluster can't silently get GPU values (Bugbot) (#616) The GPU gate enables --gpus=all + GPU chart values in Step 2, before New-K3dCluster runs. On a re-install that REUSES an existing cluster (rather than recreating it), a cluster first built in CPU mode keeps its stock rancher/k3s node -- no NVIDIA runtime, advertises 0 GPUs, no `nvidia` RuntimeClass -- and k3s's node image is fixed at create time. Writing GPU values against it would strand every experiment Pending: exactly the #616 failure this PR removes. Reconcile in the reuse path: Confirm-ReusedClusterGpuCapable inspects the reused server node's image (bounded docker inspect, mirrors Test-K3sVersionDrift) and, when GPU was requested but the node isn't the custom k3s-CUDA image, disables GPU for this run (CPU fallback stays safe) and tells the user to recreate the cluster to enable GPU. The healthy fast-path already exits before the GPU gate, so it can't wrongly enable GPU. Pure Test-NodeImageGpuCapable classifier is unit-tested (stock/CUDA/mirror/ empty); the bounded inspect + clear-flag + call-site are source-guarded. Full Pester 492 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): build the GPU node image locally so GPU installs need no login (#616) Restores the one-command promise for GPU. The GPU node image was published private, so enabling GPU forced the user to supply a registry token -- exactly the friction #612 (CPU) never had, because every image it pulls is public. Default GPU path is now a LOCAL build: on an NVIDIA machine the installer runs `docker build` from PUBLIC bases only (rancher/k3s + NVIDIA's public nvcr.io CUDA base + the public NVIDIA container toolkit), tags it locally, and creates the cluster with that image -- no registry login, no private package, one command. Build-GpuNodeImage is idempotent (reuses an already-built image), bounded with a visible progress bar (Wait-ProcessWithDeadline), sanity-checks that the built image runs k3s, and on any failure falls back to CPU with a clear reason. The Dockerfile + device-plugin manifest are embedded (self-contained verified artifact) and a Pester drift guard keeps them byte-identical to docker/k3s-cuda/*. An explicit prebuilt image / mirror (TRACEBLOC_K3S_CUDA_IMAGE or TRACEBLOC_IMAGE_REGISTRY, i.e. air-gap tenants) still PULLS instead of building. CPU is untouched: the whole block is behind the NVIDIA gate + the docker-run probe, so a CPU machine builds nothing and gets stock rancher/k3s exactly as before. Windows-installer only; staging/prod (Linux + chart) are not affected. Full Pester 499 pass; check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): 5 GPU edge cases from Bugbot (latest, reuse, preflight, fallback) (#616) Round-3 Bugbot findings on the GPU installer, all CPU-safe: - Healthy reinstalls skip the GPU fix (Med): the completed+healthy fast path exited before any GPU reconciliation, so a cluster whose values request GPU on a CPU-only node kept stranding jobs while looking healthy. Add Test-HealthyClusterGpuConsistent to the fast path -- warns + recreate remedy. - K8S_VERSION=latest bypasses the CUDA image (Med): with latest, cluster-create adds no --image (stock node) but GPU stayed enabled -> stranded jobs. Refuse GPU for latest/empty K8S_VERSION at the gate (CPU fallback + reason). - Custom GPU image overrides fail reuse (Med): the reuse check only matched `k3s-cuda:`, downgrading a renamed/digest mirror image. Test-NodeImageGpuCapable now also accepts an exact match against the configured $K3S_CUDA_IMAGE. - Optional GPU build could abort the install (Med): Build-GpuNodeImage created the build-context dir OUTSIDE its try, so a temp/AV/disk error hit the fatal trap. Moved dir creation inside try + added a catch -> CPU fallback. - nvcr.io missing from preflight (Low): Find-Gpu now runs before Test-Preflight, which probes nvcr.io + nvidia.github.io when an NVIDIA GPU is present and we'll build. SOFT (warn, not hard-fail) since GPU degrades to CPU. +10 tests; full Pester 509 pass; check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): base64-embed the GPU build inputs so host lints don't trip (CI green) The raw here-string embed of docker/k3s-cuda/* broke two host-side guards that scan install-k8s.ps1: - check-style.sh flagged `apt-get install ... curl` (curl as a PACKAGE name) as a bare-curl call. - install.Tests.ps1's "no non-ASCII in string literals" test flagged the em-dashes in the embedded Dockerfile/manifest comments (an existing test, not to be rewritten). Embed the two files as base64 instead (pure ASCII, no `curl` token), decode them to the build context at build time. Same self-contained + drift-guarded property (the Pester drift test now decodes and compares to docker/k3s-cuda/*), with nothing for the container-vs-host lints to catch. No behaviour change to the build itself. Ran the FULL scripts/tests dir this time (both install.Tests.ps1 and install-k8s.Tests.ps1): 541 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): mirror-aware GPU probe + preflight the configured GPU registry (Bugbot) (#616) Two air-gap/mirror findings: - Mirror GPU path still hit Docker Hub (Med): the GPU passthrough probe pulled nvidia/cuda from Docker Hub regardless of the mirror. Add $CUDA_PROBE_IMAGE, re-homed onto TRACEBLOC_IMAGE_REGISTRY when set, and use it in Confirm-DockerGpu -- so a mirrored/air-gapped GPU install doesn't fall back to CPU on a blocked Docker Hub. (The device-plugin image baked into a PREBUILT node image is the mirror operator's build-time concern; the installer only pulls that image.) - Custom GPU registry bypassed preflight (Low): the new nvcr.io probes were added only on the build path, but the pull path contacts the configured registry. Add an else-branch that probes the $K3S_CUDA_IMAGE registry host (when it's a real host, not a bare Docker Hub repo) so an unreachable custom/mirror registry is surfaced at preflight, not at pull time. All GPU probes stay SOFT (warn -> CPU fallback), never hard-failing a CPU-capable install. Full scripts/tests: 543 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): regenerate embedded GPU build inputs from the merged Dockerfile (#616) Merged develop, which now carries #639's k3s-CUDA Dockerfile fix. PR CI builds the branch MERGED with develop, so the drift test compared the embedded base64 (generated from #633's older Dockerfile) against develop's newer one and failed. Regenerated the base64 embed from the current docker/k3s-cuda/Dockerfile + device-plugin manifest so the embed matches the merge tree. Full scripts/tests: 543 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): GPU registry login before the probe + correct Docker Hub host (Bugbot) (#616) Two mirror/auth findings from the mirror-aware probe change: - Private mirror probe ran before login (Med): with an authenticated TRACEBLOC_IMAGE_REGISTRY, Confirm-DockerGpu pulled the mirror-hosted CUDA probe image BEFORE Confirm-GpuImagePullable did `docker login`, so a credentialed install was rejected and fell back to CPU. Extracted Connect-GpuRegistry and call it in the gate BEFORE the probe (and reuse it inside Confirm-GpuImagePullable). - Docker Hub overrides logged into the wrong host (Low): for a bare `owner/image` ref, the login host was `owner` instead of Docker Hub. Added the pure Get-RegistryHost: a segment is a registry only if it has a '.'/':' or is 'localhost'; otherwise login targets docker.io. Full scripts/tests: 550 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): reconcile GPU on adopted-reuse + re-verify a reused GPU image (Bugbot) (#616) - Adopted releases retained stale GPU requests (Med): the adopted-reuse path uses helm --reuse-values, so an older release's GPU_REQUESTS/GPU_LIMITS survived even after cluster reconciliation cleared K3D_GPU_FLAG -> stranded jobs. Moved the GPU value decision before the adopted/fresh split and pass the three GPU env keys as --set-string on the adopted upgrade, so both paths reconcile GPU to this run. - Failed GPU images were reused unchecked (Med): Build-GpuNodeImage treated any existing tag as ready. A build that completed but failed the k3s sanity check left a broken image behind that the next run would reuse. Extracted Test-GpuImageRunsK3s and run it on the reuse path too; a broken existing image is rebuilt, not reused. Full scripts/tests: 554 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): node is authoritative for GPU + login to all GPU registries (Bugbot) (#616) - Mirrored installs could keep GPU on with a dead device plugin (Med): the device-plugin image is baked as nvcr.io/... and on a blocked mirror/air-gap network the DaemonSet never becomes ready, so the node advertises 0 GPUs while GPU requests stayed active -> stranded jobs. Confirm-GpuNode now makes the NODE authoritative: after the allocatable-GPU wait, a 0 count clears K3D_GPU_FLAG (before Install-ClientHelm writes values) -> CPU fallback with a clear reason. - Combined registry overrides logged into the wrong host (Med): with TRACEBLOC_K3S_CUDA_IMAGE and TRACEBLOC_IMAGE_REGISTRY on different hosts, Connect-GpuRegistry logged into the node-image host while the probe pulled from the mirror host -> probe rejected, GPU needlessly disabled. It now logs into every DISTINCT host across the node image and the probe image. (The earlier "Static analysis" red was a transient PSGallery download corruption installing PSScriptAnalyzer -- infra flake, cleared by re-running CI; no code cause.) Full scripts/tests: 557 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): sanity-check pulled GPU images + full pull-path preflight (Bugbot) (#616) - Pulled node images bypassed the sanity check (Med): Confirm-GpuImagePullable accepted any successful pull, so a mis-tagged/broken mirror image enabled GPU and then aborted k3d cluster-create. It now runs Test-GpuImageRunsK3s on the pulled image (symmetric with the local build) -> CPU fallback on a broken image. - GPU preflight missed required hosts (Med): the pull-path preflight probed only $K3S_CUDA_IMAGE's host. nvcr.io (the baked device-plugin image, pulled at runtime on BOTH paths) is now probed whenever GPU is enabled, and the pull path probes every DISTINCT host across the node image AND the probe image (they can differ when both overrides are set). All SOFT (warn -> CPU fallback), never a hard fail. Full scripts/tests: 559 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): a null docker-build exit code is not a failure (Bugbot) (#616) With redirected stdout/stderr, $proc.ExitCode can remain $null after the process exits (even though Wait-ProcessWithDeadline calls WaitForExit()), so `$proc.ExitCode -ne 0` misclassified a SUCCESSFUL GPU build as failed and silently dropped GPU. Mirror the #611 k3d-create idiom: fail only on a CONFIRMED non-zero exit; a null code defers to the k3s sanity check (Test-GpuImageRunsK3s), which is the authoritative "did the build produce a working image" success marker. A null exit + working image now enables GPU; a null exit + broken image still falls back to CPU. Full scripts/tests: 561 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): GPU node-verification + fallback correctness (Bugbot) (#616) Four findings surfaced on the develop-merge review: - GPU verification always read zero (High): Confirm-GpuNode parsed the whole allocatable map (`jsonpath='{...allocatable}'` renders Go's map[...] with no JSON quotes), so the "nvidia.com/gpu" regex never matched -> every healthy GPU read as 0. Combined with the new 0-count fallback that reverted ALL GPU installs to CPU. Fixed: target the scalar field directly (jsonpath '...allocatable.nvidia\.com/gpu'). - Plugin failure left GPU requests enabled (High): when Install-GpuDevicePlugin returned false, K3D_GPU_FLAG stayed set and Confirm-GpuNode was skipped, so the chart still requested GPUs -> stranded jobs. Now a real plugin failure (flag still set) clears the flag -> CPU fallback. - K8S_VERSION=latest retained the flag (Med): the latest branch recorded a reason but didn't clear K3D_GPU_FLAG that Install-NvidiaContainerToolkit may have set -> a stock 'latest' cluster could get --gpus=all without the CUDA image. Now cleared. - Fast-path read stale GPU values (Low): Test-HealthyClusterGpuConsistent read local values.yaml, which is only clientId-healed on the adopted path. Now reads the LIVE Helm release values (helm get values) instead. Full scripts/tests: 564 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): disable NVIDIA_REQUIRE_CUDA so an older-but-valid driver enables GPU (#616) Real-box test (RTX 4050, driver 532.10 = CUDA 12.1) showed GPU falling back to CPU. Root cause: the CUDA base image bakes NVIDIA_REQUIRE_CUDA=cuda>=12.4, and the NVIDIA container runtime REFUSES to start the container when the driver is older ("unsatisfied condition: cuda>=12.4"). So a perfectly good GPU on a slightly older driver read as "Docker can't expose the GPU" and dropped to CPU. The probe and the k3s node run nvidia-smi / k3s, NOT CUDA workloads -- the real CUDA/driver compatibility for training is enforced per-pod by each training image -- so this requirement gate is meaningless for us. Disable it: - Confirm-DockerGpu probe: docker run ... -e NVIDIA_DISABLE_REQUIRE=1 - node image: ENV NVIDIA_DISABLE_REQUIRE=1 (so the k3s node boots under --gpus on an older driver); base64 embed regenerated. Now GPU enables on any driver that can actually pass the GPU through, regardless of the base image's CUDA version. Full scripts/tests: 566 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): key GPU-image reuse on build-input content hash, not just the tag (Bugbot) (#616) Adding NVIDIA_DISABLE_REQUIRE didn't change the image tag, and Build-GpuNodeImage reused any cached image that merely printed a k3s version -- so a stale pre-fix image would be reused and still fail cluster-create on an older driver. Stamp a short content hash of the build inputs (embedded Dockerfile + device-plugin manifest) as a `tracebloc.k3s-cuda-content` label at build time, and on reuse require that label to match the CURRENT hash (plus the k3s sanity check). Any change to the build inputs now busts the cache -> a stale image is rebuilt, not reused. Generalizes beyond this one fix to any future Dockerfile change. Full scripts/tests: 568 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): fast-path retries GPU + image sanity check exercises --gpus (Bugbot) (#616) - Healthy fast path prevented GPU retries (Med): a completed CPU-fallback install exited "nothing to do" before Find-Gpu/probe, so re-running after fixing Docker/driver/registry could never enable GPU -- contradicting the fallback guidance. Find-Gpu now runs before the fast path, and when an NVIDIA GPU is present but the running cluster is CPU-only (Test-RunningClusterGpuCapable) the fast path falls through to the full walk to retry GPU instead of exiting. - Image sanity check skipped GPU startup (Med): Test-GpuImageRunsK3s ran without --gpus, so a stale mirror image lacking NVIDIA_DISABLE_REQUIRE passed the check and then failed k3d cluster-create on an older driver (aborting instead of CPU fallback). It now runs WITH --gpus (no bypass), exercising the exact requirement gate cluster-create hits -> a stale image drops to CPU cleanly. Full scripts/tests: 571 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): fast-path GPU-live check uses allocatable GPU + bound nvidia-smi (Bugbot) (#616) - Fast path mistook a CUDA image for a working GPU (Med): Test-RunningClusterGpuCapable keyed on the node IMAGE name, but a CUDA node with a failed device plugin has 0 GPUs -- so a CPU-fallback re-run would wrongly take the fast path and never retry. It now checks the node's LIVE allocatable nvidia.com/gpu (the same authoritative signal Confirm-GpuNode uses), bounded with --request-timeout. - Unbounded nvidia-smi in Find-Gpu (Med): Find-Gpu now runs before the fast path, and its driver/VRAM nvidia-smi probes had no deadline, so a wedged driver could hang every re-run. Extracted Invoke-BoundedProcess (generic bounded+killable child; Invoke-DockerCli is now a thin wrapper over it) and Confirm-NvidiaDriver uses it for nvidia-smi (15s). Full scripts/tests: 572 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): fast path reconciles GPU both directions -- enable AND CPU-drift (Bugbot) (#616) The prior fix only handled "GPU present but node CPU-only -> retry". The reverse also leaked: when the node now advertises a GPU (e.g. a delayed device-plugin recovery) but the live release still requests CPU, the fast path exited "nothing to do" and training silently stayed on CPU. The fast path now shortcuts ONLY when GPU is FULLY consistent -- the node advertises a GPU AND the live release requests one (Test-RunningClusterGpuCapable + Test-LiveReleaseRequestsGpu). Any other combo on a GPU machine falls through to the full walk, which reconciles GPU_REQUESTS in either direction. Extracted the live-release GPU check into Test-LiveReleaseRequestsGpu (shared with Test-HealthyClusterGpuConsistent). Full scripts/tests: 572 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gpu): real GPU on Windows/WSL2 via CDI (validated on hardware) (#616) Validated end-to-end on a live RTX 4050 laptop (driver 532.10, Docker Desktop/WSL2): a pod that REQUESTS nvidia.com/gpu now schedules AND runs CUDA compute (vectorAdd "Test PASSED"). Previously GPU always fell back to CPU on Windows. Why the old approach couldn't work, and what replaces it: * The NVIDIA k8s DEVICE PLUGIN cannot work on Docker Desktop/WSL2: the GPU is paravirtualized (/dev/dxg + a WSL driver store), so nvmlInit() returns ERROR_NOT_SUPPORTED and the plugin registers 0 GPUs. Because it OWNS the nvidia.com/gpu extended resource it also pins the node at 0, stranding jobs. -> the node image no longer ships it (RuntimeClass only), and the installer advertises nvidia.com/gpu itself via a node-status patch (Set-NodeGpuCapacity). * CUDA itself works fine through CDI. The node image now runs nvidia-container-runtime in CDI mode and generates the WSL CDI spec AT BOOT (tracebloc-cdi-boot.sh) -- it must be at boot because the driver-store path contains a per-machine hash. * Critical gap found: `nvidia-ctk cdi generate --mode=wsl` OMITS libdxcore.so (it lives in the standard lib path, not the driver store). Without it libcuda loads but can't reach /dev/dxg, surfacing as a misleading "CUDA driver version is insufficient for CUDA runtime version". The boot script injects it. CPU and Linux are unaffected: the boot script is a strict no-op without /dev/dxg, the installer only takes the CDI path when /dev/dxg exists in the node (otherwise the standard device-plugin path is unchanged), and every step failure-falls back to CPU. Follow-up (separate repo/PR): jobs-manager must set NVIDIA_VISIBLE_DEVICES=nvidia.com/gpu=all on GPU training pods. +8 tests; full scripts/tests 580 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): node self-heals GPU capacity + verify the CDI spec (Bugbot) (#616) - GPU capacity lost after node restart (HIGH): nvidia.com/gpu was advertised by a one-shot install-time status patch, but a manually patched extended resource is NOT durable -- the kubelet re-reports node status on start, so a Docker Desktop or Windows restart dropped it while the chart still requested a GPU, stranding every job Pending until the installer was re-run. There's no device plugin to own the resource on WSL2, so the NODE now re-asserts it: the boot script (the entrypoint, so it runs on every start) backgrounds a reconciler that waits for the local API and re-patches whenever the capacity is missing/0. A reboot self-heals with no user action. Interval override: TRACEBLOC_GPU_RECONCILE_SECS. - CDI success wasn't verified (MED): the WSL2 path printed "GPU acceleration enabled" after only the capacity patch. Since the boot script guards every step with `|| true`, a failed `nvidia-ctk cdi generate` would leave us advertising a GPU pods can't use -- jobs schedule then fail CUDA with no cluster-level signal. Now the installer verifies /etc/cdi/nvidia.yaml exists and is non-empty in the node before claiming success, else CPU fallback with a clear reason. Both stay CPU-safe: the reconciler is backgrounded + fully guarded (can never delay or block k3s) and only runs when /dev/dxg exists; the spec check failure-falls back to CPU. +2 tests; full scripts/tests 582 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gpu): chart wiring for the CDI device selector + path-correct GPU guidance (#616) Completes the GPU-on-Windows chain: the installer now tells the chart to thread the CDI device selector into GPU training pods, which jobs-manager consumes as NVIDIA_VISIBLE_DEVICES (tracebloc/client-runtime#291). Without this the node advertises a GPU and pods schedule, but CUDA fails inside them. Chart: documented env.GPU_VISIBLE_DEVICES in values.yaml + values.schema.json. No template change needed -- jobs-manager-deployment's generic env pass-through already forwards it to both containers (verified with `helm template`: present when set, absent when unset/empty). Chart 1.9.27 -> 1.9.28 (version + appVersion in lockstep). Installer: $GPU_DEVICE_SELECTOR is set ONLY on the WSL2/CDI path and written as env.GPU_VISIBLE_DEVICES (also forced via --set-string on the adopted-reuse reconcile, so a stale value can't survive). Empty on a normal device-plugin (Linux) node, where the plugin owns NVIDIA_VISIBLE_DEVICES and sets concrete GPU UUIDs -- forcing a CDI selector there would break device resolution. Also fixes two misleading-guidance findings (Bugbot): - The 0-GPU skip reason always blamed the NVIDIA device plugin and told operators to mirror nvcr.io/nvidia/k8s-device-plugin. On the WSL2/CDI path there IS no device plugin, so that was a dead end. Now branches per path. - The doctor's GPU smoke-test command omitted runtimeClassName: nvidia (pods only get the GPU under it), so running it on a WORKING cluster looked like failure. It now carries the RuntimeClass, and on WSL2/CDI suggests a CUDA workload instead of nvidia-smi -- verified on real hardware, nvidia-smi fails in a pod on WSL2 (NVML unsupported) even when CUDA compute works. +4 tests; full scripts/tests 586 pass / 0 fail; check-style, check-facts, helm lint green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): remove a leftover device plugin, correct the CDI cause, collapse SERVERS (Bugbot) (#616) - CDI path left the device plugin fighting (HIGH): a device plugin OWNS the nvidia.com/gpu extended resource and on WSL2 re-reports 0 on every sync, so a DaemonSet left behind by an older install (or by a run where the /dev/dxg probe transiently missed and we took the plugin path) would overwrite the capacity patch and keep GPU disabled across ALL re-runs. The CDI path now deletes it first (--ignore-not-found, so the normal case is a clean no-op) before advertising. - Wrong CPU-fallback reason on CDI (MED): when Set-NodeGpuCapacity failed, no GPU_SKIP_REASON was set, so the caller's generic fallback filled in a device-plugin failure -- the wrong cause on a path that never uses the plugin. Set it here. - GPU mode ignored SERVERS (MED): AGENTS was collapsed to 0 but SERVERS was left alone, and EVERY server node runs the boot reconciler and advertises nvidia.com/gpu=1 for the SAME physical card -- so SERVERS>1 offered N GPUs for one device. GPU mode now forces servers=1 too, overriding an explicit value loudly. +3 tests; full scripts/tests 589 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): ship CDI setup as a k3d entrypoint DROP-IN -- the image ENTRYPOINT never ran (#616) Root cause of "GPU couldn't be wired into the cluster (CDI spec missing)" on a real Windows box. k3d does NOT use the node image's ENTRYPOINT: it replaces it with its own /bin/k3d-entrypoint.sh, which runs every /bin/k3d-entrypoint-*.sh drop-in and then execs k3s. Confirmed on the live node: Entrypoint = ["/bin/k3d-entrypoint.sh"] /bin/k3d-entrypoint-{cgroupv2,dns,mounts}.sh <- k3d's own drop-ins /usr/local/bin/tracebloc-cdi-boot.sh <- ours, never invoked /etc/cdi <- missing (spec never generated) So the CDI spec was never created and the installer correctly fell back to CPU (the verification added in c37e133 is what surfaced this instead of silently "succeeding"). Fix: install the script as /bin/k3d-entrypoint-tracebloc-cdi.sh -- k3d's documented extension point -- and restore the stock ENTRYPOINT ["/bin/k3s"]. As a drop-in it must RETURN rather than exec k3s (k3d does that), and must always exit 0: k3d runs drop-ins with `|| exit 1`, so a non-zero exit would abort the whole node, and GPU is optional. The capacity reconciler is now fully detached (</dev/null, output to /dev/null) since the drop-in exits immediately after forking it. Also corrected the Dockerfile header, which still claimed the image bakes in the device plugin (it ships only the RuntimeClass now). +2 regression tests (drop-in path + stock ENTRYPOINT; returns and never execs k3s); full scripts/tests 591 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): pass the node capacity patch via --patch-file (PS 5.1 ate the JSON quotes) (#616) Root cause of "Couldn't advertise GPU capacity on the node" on a real Windows box. Windows PowerShell 5.1 does not preserve embedded double quotes when it builds a native command line, so kubectl patch node N --type=json -p '[{"op":"add","path":...,"value":"1"}]' reached kubectl as [{op:add,...}] -> invalid JSON -> the patch failed on EVERY run. The same command works by hand only because each quote is escaped as \" there. (This is why the CDI spec now generates fine but capacity never appeared.) Fix: write the patch to a temp file and use --patch-file, which sidesteps the shell entirely. UTF8 without BOM (a BOM breaks kubectl's JSON parse). Also retried 6x/5s: the node object can still be settling right after cluster-create, and a transient 404 there would drop an otherwise-working GPU to CPU for the whole run. Temp file always cleaned up. +1 regression test (asserts --patch-file, no inline -p, BOM-less encoding, retry loop); full scripts/tests 592 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): print the doctor's GPU test with escaped quotes so it can actually be pasted (#616) Same PowerShell quote-stripping class as the capacity patch, now in user-facing guidance: the doctor printed `--overrides='{"spec":...}'`, which dies with "error: Invalid JSON Patch" when pasted into Windows PowerShell (confirmed on a live box). A suggested command that can't be run is worse than none, so the JSON is now emitted with every quote escaped as \" -- the form proven to work by hand. Verified the rendered string is paste-able: --overrides='{\"spec\":{\"runtimeClassName\":\"nvidia\", ... }}' full scripts/tests 592 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): libdxcore injection missed the generator's indentation (real CUDA failure) (#616) Root cause of "CUDA driver version is insufficient" inside GPU pods on a real box. Confirmed live: the CDI spec existed and parsed (`nvidia-ctk cdi list` -> 1 device), but `grep libdxcore /etc/cdi/nvidia.yaml` found NOTHING -- the injection never happened. The awk anchor was `^ mounts:$` (exactly two spaces), while nvidia-ctk marshals YAML with yaml.v3 at FOUR spaces, so it never matched. Without libdxcore, libcuda loads but can't reach /dev/dxg -> that misleading driver error, on a node that advertises a healthy GPU. Fixes: - Indent-agnostic anchor (`^[[:space:]]*mounts:[[:space:]]*$`) and the inserted item now MIRRORS the generator's own first-item indentation instead of hardcoding it. Hardcoding was doubly wrong: YAML forbids mixing indents in one list, so a fixed 4-space item next to 2-space items would have made the whole spec unparseable (CDI then injects nothing -- same symptom, different cause). Verified both 2- and 4-space specs parse with libdxcore present, and that the old awk produced invalid YAML on a 2-space spec. - The edit is only adopted if `nvidia-ctk cdi list` still parses the result; otherwise the original spec is restored (GPU without libdxcore beats a silently disabled GPU). - The installer now verifies libdxcore is IN the spec (not merely that a spec exists) and falls back to CPU with a specific reason if not -- so this class of silent miss surfaces at install time instead of as a confusing CUDA error later. +2 regression tests; full scripts/tests 594 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): never advertise a GPU that CDI can't back, + honest GPU messaging (Bugbot) (#616) - Reconciler advertised GPU without CDI (HIGH): the boot reconciler re-asserted nvidia.com/gpu whenever /dev/dxg existed, even if `cdi generate` failed or left an incomplete spec. The installer already refuses in that case, but a Docker Desktop or Windows restart re-ran the reconciler and put capacity back onto a node where injection is broken -- pods schedule, then fail CUDA with no cluster-level signal. The reconciler now applies the SAME standard (cdi_ok: spec non-empty AND contains libdxcore AND `nvidia-ctk cdi list` parses) and simply doesn't start otherwise. - libdxcore path was fragile (MED): hardcoded /usr/lib/x86_64-linux-gnu/libdxcore.so, but it lives elsewhere on other Docker Desktop / WSL2 versions (/usr/lib/wsl/lib, the WSL driver store). A miss silently skipped the injection while the spec still passed the non-empty check -> GPU advertised, CUDA fails with the misleading driver error. Now probes the known locations, falls back to the linker cache, and mounts it at the path where it was actually found. - Green success before verification (MED): `Ok "GPU acceleration enabled (WSL2/CDI)"` printed at the capacity patch, before Confirm-GpuNode confirms allocatable GPU -- so a failed verification showed a green enabled line followed by a CPU fallback. Now an Info "verifying..." line; Confirm-GpuNode's "GPU verified and available" is the only success claim. - Misleading preflight warning + stale comments (LOW): the soft GPU preflight always said the node image "can't be built", including on the pull/mirror path where nothing is built locally -- now path-aware. Also corrected three comments that still claimed the device plugin is baked into the node image (this PR ships only the RuntimeClass). +4 tests; full scripts/tests 598 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): the in-WSL toolkit step no longer claims GPU is ready (Bugbot) (#616) Install-NvidiaContainerToolkit printed a green "GPU acceleration ready", set K3D_GPU_FLAG, and CLEARED GPU_SKIP_REASON after merely verifying the toolkit inside the WSL distro. But Confirm-DockerGpu is the authoritative gate and runs later, so: - operators could see a green ready line and then a CPU fallback, and - clearing the skip reason dropped the real cause when the later gate failed. It now reports only what it established ("NVIDIA Container Toolkit present in <distro>", Info not Ok) and logs that GPU is still gated on the Docker GPU probe. It no longer touches K3D_GPU_FLAG or GPU_SKIP_REASON. Nice invariant this establishes: K3D_GPU_FLAG is now assigned "--gpus=all" in exactly ONE place -- the authoritative gate -- which a new test pins, along with the toolkit step no longer setting the flag or clearing the reason. +1 test; full scripts/tests 599 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(gpu): machine-check the smoke-test JSON instead of eyeballing it (Bugbot) (#616) Bugbot reported an "extra closing brace" in the WSL2/CDI --overrides payload. Verified FALSE POSITIVE two ways: a JSON parser accepts it with the correct structure, and this is the exact escaped form that ran on a live box (kubectl created the pod; it failed later at CUDA, not at parsing). The `}}}` after "1" is correct -- it closes limits, then resources, then the container object. No behaviour change, but the concern deserves to be checkable rather than argued: extracted the command into a PURE Get-GpuSmokeTestCommand (selector in, string out), so tests can render both variants, strip the paste-escaping and ConvertFrom-Json them. A genuine brace slip now fails CI instead of shipping a command that errors on a healthy cluster. Also removes the duplication between the two branches and gives the diagnostics bundle one source of truth. +2 tests (JSON validity for the WSL2/CDI and device-plugin variants), and the previous source-scraping assertions now go through the builder. Full scripts/tests 601 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): close version-drift gaps that would cost GPU on a different Windows machine (#616) Audit prompted by "what breaks on someone else's laptop". The reassuring part first: a version mismatch CANNOT break the install. The k3d drop-in runs under `|| exit 1`, so a failure there would abort the node and fail cluster-create -- I verified under real `dash` (the node image's /bin/sh, no bashisms) that it exits 0 with the toolkit absent, with every nvidia-ctk subcommand failing, and with a garbage spec. It degrades to CPU instead. Two gaps found that would silently LOSE GPU on another machine: 1. The NVIDIA Container Toolkit was UNPINNED (`apt-get install nvidia-container-toolkit`), so two machines built weeks apart could get different builds. The whole WSL2 path leans on version-sensitive surfaces -- `cdi generate --mode=wsl`, `config --set nvidia-container-runtime.mode=cdi`, and the exact YAML the generator emits (our libdxcore injection parses it) -- so a future release changing the spec shape would break GPU on new installs while existing ones kept working. Pinned to the build validated on hardware (1.19.1-1) via ARG NCT_VERSION, and it FALLS BACK to latest if that version has aged out of the apt repo, so a stale pin degrades to "unpinned" rather than failing the build (which would cost GPU entirely). Also logs `nvidia-ctk --version` so the image records what it got. 2. The cdi_ok gate required `nvidia-ctk cdi list` to EXIST. That subcommand is version-dependent, so on a toolkit build without it we would have refused to advertise a perfectly working GPU -- a false negative on someone else's machine. The gate now uses format-stable structural facts (spec non-empty, declares nvidia.com/gpu, exposes /dev/dxg, carries the libdxcore mount) and uses `cdi list` only as an EXTRA veto when available. Verified both directions with a dash harness: a good spec + a toolkit lacking `cdi list` now advertises the GPU; a spec missing libdxcore still does not. +2 tests; full scripts/tests 603 pass / 0 fail; dash -n, check-style, check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gpu): every GPU-enable failure now names an actionable cause (#616) The residual risks that differ machine to machine all degraded to CPU correctly, but most collapsed into "the GPU node image build failed (docker build exit 1)" -- true, and useless to the person holding the laptop. Now each one names what happened and the single thing to do about it: - older Docker Desktop / no BuildKit labs frontend -> "update Docker Desktop, or point TRACEBLOC_K3S_CUDA_IMAGE at a prebuilt image" - full disk -> "free up space and re-run" - retired CUDA base tag -> names the tag + TRACEBLOC_CUDA_BASE_TAG - TLS-inspecting proxy (x509) -> TRACEBLOC_CA_BUNDLE - blocked/offline registry -> TRACEBLOC_IMAGE_REGISTRY (the air-gap path) - registry rate limit -> retry or use the mirror - anything else -> exit code + a pointer to the build output in the log Classification lives in a PURE Get-GpuBuildFailureReason so every branch is unit-tested. Also: - The Docker GPU probe failure now quotes the DETECTED driver version and a concrete minimum ("update to 525 or newer"). Our install gate accepts 460+, but WSL2 CUDA needs much newer, so "update your driver" left people unable to tell if theirs qualified. - GPU-detected-but-not-enabled gets its OWN summary block instead of being crammed into the Mode line: what happened, why, "fix that then re-run", and the log path. Previously the reason was a long parenthetical that was easy to miss. +12 tests; full scripts/tests 615 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): no green line may claim GPU is enabled before verification (Bugbot) (#616) Third report of this defect, so I swept for the whole family instead of patching one line. Fixed the reported one: the top-level gate printed Ok "GPU enabled -- cluster will use the custom k3s-CUDA image with --gpus=all" at Step 2, before cluster-create, the node's CDI wiring, and Confirm-GpuNode -- any of which can still clear K3D_GPU_FLAG. Now Info, worded as intent ("GPU support prepared … verified once the node is up"). The sweep found TWO MORE instances on the device-plugin path (pre-existing, not introduced here, but the same defect and the same misleading sequence): Ok "GPU acceleration enabled." both when the DaemonSet already existed and after a successful rollout -- printed before Confirm-GpuNode, which can find 0 allocatable GPUs and fall back to CPU. Both are now Info lines that say verification is pending. Message-only; no behaviour change on that path. Established the invariant instead of relying on review: Ok is reserved for facts already established, Info for intent. Exactly two Ok-level GPU lines remain, and both state something verified -- the image was built AND passed its k3s sanity check, and the node WAS observed advertising a GPU. A test pins the count at 2, asserts neither says "enabled", and asserts the four pre-verification lines are Info. +3 tests; full scripts/tests 618 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): the CDI revert must not require `cdi list` either (Bugbot) (#616) My own inconsistency, caught correctly. I made the cdi_ok gate tolerate a missing `nvidia-ctk cdi list` (it's version-dependent), but left the REVERT one block earlier calling it unconditionally. On a toolkit without that subcommand the sequence was: libdxcore injected -> `cdi list` "fails" (absent) -> edit reverted -> spec now lacks libdxcore -> installer reports "the spec is missing libdxcore" -> CPU i.e. a perfectly good injection thrown away, and a reason that is factually wrong (the library WAS found and mounted) and unactionable. The revert now reverts only when the parser is AVAILABLE and actively REJECTS the result -- the same rule as cdi_ok. Verified with a dash harness: with `cdi list` absent, libdxcore survives in the spec, the spec is still valid YAML, and the reconciler advertises the GPU. +1 test asserting every `cdi list` use is availability-gated: two probes, exactly two executable bare uses (comments excluded), and both guarded shapes pinned. Full scripts/tests 619 pass / 0 fail; dash -n, check-style, check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(gpu): shell-runnable drift guard for the embedded node-image files (review #633) Review asked for a check-facts/bats-style guard on the base64 embeds, having verified they are byte-identical today but noting nothing enforces it: check-facts guards the version PINS, not whole-file embedding, and Get-GpuBuildContentHash only hashes the EMBEDDED copies for local-build caching -- it never compares them to the repo files. A future edit to docker/k3s-cuda/* that isn't re-embedded would silently diverge (published image vs the installer's local build) with nothing failing. A Pester suite already asserts exactly this and runs in CI on windows-latest AND ubuntu-latest, so drift was in fact caught -- the reviewer simply had no pwsh to see it. That is a fair ergonomics gap, so this adds the bash-side mirror: decode each of the three blobs and diff against docker/k3s-cuda/*, plus non-empty checks (a truncated re-embed) and the drop-in shape invariant (no `exec /bin/k3s`, always `exit 0` -- the mistake that shipped once). CI already runs `bats scripts/tests/*.bats`, so it is picked up automatically. Verified the guard actually bites: appending a line to nvidia-runtimeclass.yaml without re-embedding turns test 2 red, and restoring it returns 5/5 green. On failure it prints the diff and the exact remedy (re-embed + regenerate the manifest). bats-hygiene passes (every assertion carries `|| return 1`, per #527). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): quote whitespace args + fail fast when GPU hosts are known-blocked (Bugbot) (#616) Two of four findings were real; the other two are refuted with evidence in the threads. 1. (Med) Arguments with whitespace could split. Invoke-BoundedProcess joins args into ONE command line, so an unquoted value containing a space -- a registry username, or a temp path under a profile like "C:\Users\First Last\..." -- silently became two arguments and corrupted the command. Now quotes any arg containing whitespace, leaves already-quoted values alone (so call sites that quote themselves aren't double-quoted), and preserves an empty string as a present-but-empty argument. Fixes the class, not just docker login. 2. (Med) A re-run on a restricted network looked hung. Preflight already soft-probes the GPU download hosts, but the gate then ran the probe (180s) and the build/pull (up to 15-20 min) anyway, so the "re-run to retry GPU" advice we print led to minutes of timeouts before the CPU fallback. Preflight now records the unreachable host and the gate short-circuits on it with an actionable reason (set TRACEBLOC_IMAGE_REGISTRY / a prebuilt image). Defaults empty, so a reachable machine is unaffected. +5 tests (incl. behavioural joiner cases: space-containing value, already-quoted path, empty string). Full scripts/tests 625 pass / 0 fail; embed-drift bats 5/5; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): only path-required hosts may skip GPU -- don't break air-gapped mirrors (Bugbot) (#616) A regression I introduced one commit earlier. The fast-fail short-circuit treated EVERY soft GPU host probe as blocking, but on a mirror / prebuilt-image install nvcr.io is unreachable BY DESIGN -- the node image and the CDI probe image are both re-homed to the mirror. So the "don't waste minutes timing out" optimisation disabled GPU for exactly the air-gapped case the mirror exists to serve, and the remedy told the operator to configure the mirror they had already configured. Probes now carry gpuBlocking, set only for the hosts the CURRENT path actually needs: - build path: nvcr.io + nvidia.github.io -> blocking - mirror/prebuilt path: the configured GPU registry host -> blocking; nvcr.io still probed (a warning is useful) but NON-blocking Only a blocking probe arms GPU_HOSTS_UNREACHABLE. The skip remedy is now path-aware too: on the mirror path it says to check that the configured GPU registry is reachable and holds the k3s-CUDA image, rather than suggesting a mirror. +2 tests pinning both directions. Full scripts/tests 627 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…ed (#672) The rule said "for two releases". Verified against git instead: the `chown … && chmod …` shape entered in chart 1.9.20 (#611/#612, commit a07f76b) and survived every version through 1.9.33 — thirteen chart versions, not two. #667 (7852f02) rewrote the modes on that exact line and left the chain untouched, which is the more useful half of the lesson: the line was re-read for its modes and not for its control flow. Also corrects the issue's attribution of the chain to #667. Refs #672, #667, #611 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… fails (#672) (#689) * fix(chart): stop init-writable-data skipping the chmod when the chown fails (#672) init-writable-data ran `chown 1000:1000 "$d" && chmod "$m" "$d" || echo …`, so a refused chown short-circuited the chmod and the mode was never applied — while the message said "leaving as-is", implying nothing could be done. That inverts the priority. kubelet ignores fsGroup on hostPath (kubernetes/kubernetes#138411), so the MODE is what makes these trees usable: /data/shared must be other-writable for the ingestion Job (uid 65534, or HOST_UID) and the CLI staging/teardown pod (uid 65532), neither of which is 1000 nor shares a group with it. The chown is cosmetic next to that, and it is also the call most likely to be refused — on a Windows/Docker-Desktop bind mount or an NFS root_squash export it is precisely what fails. So the failure that mattered least was cancelling the one that mattered most, silently nullifying the 2777/3777 split from #667 on the platform that split was written for. Symptom: #653's `mkdir: can't create directory '/data/shared/.tracebloc-staging/': Permission denied`. The chown and the chmod are now separate best-effort statements, each recording whether it failed, and the per-dir verdict is graded on the mode OBSERVED afterwards via `ls -ldn` rather than on either exit status — a bind mount can accept a chmod and ignore it, so an exit code is not evidence. A partial result is reported as such ("chown failed; mode applied anyway") instead of implied. Unchanged: per-dir modes, per-dir independence, non-fatal behaviour, POSIX sh for busybox. Kept diffable by eye against the installer's Get-ReleaseDirsPrepCommand, which already does it this way. Verified by executing the helm-rendered command[2], not by reading it: - sh -n, dash -n, bash --posix -n all clean - busybox:1.35 as root: /data/shared drwxrwsrwx, /data/logs drwxrwsrwt, exit 0 - busybox:1.35 with --cap-drop CHOWN (chown refused, chmod permitted): modes STILL land drwxrwsrwx / drwxrwsrwt; the old command leaves both at drwxr-xr-x - /data/shared read-only (both calls fail): FAIL reported with the real errno, /data/logs still fixed, exit 0 - end-to-end on a shared volume after a refused chown: uid 65534 creates .tracebloc-staging and writes /data/logs; uid 65532 unlinks uid 65534's entries in /data/shared (no sticky) but not in /data/logs (sticky) — both splits intact Tests: the new #672 case fails against the old command and passes against the fix. The obvious comment-scoped guard (`^[^#\n]*chown.*&&.*chmod`) is silently VACUOUS — `${e#*:}` puts a '#' before the chown — so the guard is unscoped and the template describes the old shape in words instead. Existing assertions kept, updated for the multi-line command. jobs_manager_test.yaml 34 -> 35 passing; full suite 379 -> 380 passing with develop's 5 failed / 5 errored baseline unchanged. Also adds the recurring-finding rule to .cursor/BUGBOT.md per CLAUDE.md. Refs #672, #667, #653, #654 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): report only what init-writable-data actually observed (#672) Two overclaims in the first commit's own reporting, both found by running the failure paths rather than reading them — the same family as the bug being fixed. 1. The verdict grades other-writability alone (correctly: that is what decides whether uid 65534/65532 can work, and failing a setgid-stripped-but-writable mount would cry wolf on a working install). But it labelled that bare "OK", which reads as "the whole mode landed". Now says "OK <dir> other-writable" and always prints want vs got, so a mount that granted other-write while dropping S_ISGID is visible instead of implied. 2. Worse: the partial-result note said "mode applied anyway" whenever any call failed. On a dir that was ALREADY other-writable and where BOTH calls were refused, that is simply false — nothing this container did applied anything. Reproduced in busybox:1.35 (pre-set 1777, run as a non-owner uid so chown and chmod are both refused): want 2777 got drwxrwxrwt uid 0 (chown+chmod failed; mode applied anyway) Now reads "(chown+chmod failed; other-writable regardless)" — it claims the observation, not a causal link it cannot support. Re-verified on the helm-rendered command[2]: sh -n / dash -n / bash --posix -n clean; root happy path lands drwxrwsrwx + drwxrwsrwt; chown-refused still lands both modes; already-1777 with both calls refused now reports truthfully; read-only /data/shared still FAILs with the real errno while /data/logs is still fixed; exit 0 throughout. Tests pin both strings, including a notMatchRegex on the old "mode applied anyway" wording. 35 passing, full suite 380 with develop's 5 failed / 5 errored baseline unchanged. Refs #672 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(chart): move the verdict rationale out of the container command (#672) The long "why other-writability alone is the pass condition" prose was inside the script passed to `sh -c`, so it shipped in the pod spec and showed up in every `kubectl get deploy -o yaml`. It belongs in the YAML comment above, which does not. Left a two-line pointer where a script editor will see it. Also records the one intentional divergence from the installer's Get-ReleaseDirsPrepCommand: the chart does not redirect chown/chmod stderr to /dev/null, so the real errno (Operation not permitted vs Read-only file system) lands in `kubectl logs` next to the verdict. The installer suppresses it because its output is a user-facing progress line; an init container's log is a debugging surface, and hiding the errno there would remove the evidence a reader needs. Comment-only inside command[2]: re-rendered and re-ran the chown-refused path in busybox:1.35 to confirm byte-identical output and modes (drwxrwsrwx / drwxrwsrwt, exit 0). 35 passing; full suite 380 passing, baseline unchanged. Refs #672 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(bugbot): correct how long the chained chown/chmod actually shipped (#672) The rule said "for two releases". Verified against git instead: the `chown … && chmod …` shape entered in chart 1.9.20 (#611/#612, commit a07f76b) and survived every version through 1.9.33 — thirteen chart versions, not two. #667 (7852f02) rewrote the modes on that exact line and left the chain untouched, which is the more useful half of the lesson: the line was re-read for its modes and not for its control flow. Also corrects the issue's attribution of the chain to #667. Refs #672, #667, #611 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

shujaatTracebloc self-assigned this Aug 5, 2026

shujaatTracebloc marked this pull request as ready for review August 5, 2026 14:20

shujaatTracebloc requested a review from saadqbal as a code owner August 5, 2026 14:20

shujaatTracebloc changed the title ~~fix(installer): make the checksum drive the download retry so a truncated tool binary self-heals (#609)~~fix(installer): make the checksum drive the download retry so a truncated tool binary self-heals (#611)Aug 5, 2026

shujaatTracebloc force-pushed the fix/609-checksum-driven-download branch from 4032a15 to 436ae6eCompare August 5, 2026 14:20

shujaatTracebloc marked this pull request as draft August 5, 2026 14:21

cursorBot reviewed Aug 5, 2026
View reviewed changes

Comment threadscripts/install-k8s.ps1

shujaatTraceblocand others added 2 commits August 5, 2026 16:33

shujaatTracebloc marked this pull request as ready for review August 6, 2026 06:45

shujaatTracebloc added the pii-gate-override Bypass the public PII gate (reviewed false positive or gate cannot run); visible in audit label Aug 6, 2026

cursorBot reviewed Aug 6, 2026
View reviewed changes

Comment threadscripts/install-k8s.ps1
Comment threadscripts/install-k8s.ps1

shujaatTracebloc added pii-gate-override Bypass the public PII gate (reviewed false positive or gate cannot run); visible in audit and removed pii-gate-override Bypass the public PII gate (reviewed false positive or gate cannot run); visible in audit labels Aug 6, 2026

shujaatTraceblocand others added 3 commits August 6, 2026 09:09

saadqbal reviewed Aug 6, 2026
View reviewed changes

Comment threadclient/templates/jobs-manager-deployment.yaml Outdated
Comment threadclient/templates/jobs-manager-deployment.yaml Outdated
Comment threadscripts/install-k8s.ps1 Outdated

shujaatTraceblocand others added 2 commits August 6, 2026 10:49

Merge remote-tracking branch 'origin/develop' into fix/609-checksum-d…
d650c23
…riven-download

cursorBot reviewed Aug 6, 2026
View reviewed changes

Comment threadclient/templates/jobs-manager-deployment.yaml Outdated

shujaatTraceblocand others added 2 commits August 6, 2026 11:03

cursorBot reviewed Aug 6, 2026
View reviewed changes

Comment threadclient/templates/jobs-manager-deployment.yaml Outdated
Comment threadclient/templates/jobs-manager-deployment.yaml Outdated

saadqbal approved these changes Aug 6, 2026
View reviewed changes

shujaatTracebloc merged commit a07f76b into developAug 6, 2026
45 of 48 checks passed

shujaatTracebloc deleted the fix/609-checksum-driven-download branch August 6, 2026 09:43

divyasinghds mentioned this pull request Aug 6, 2026
fix(#558): write Docker Desktop autostart for a provisioned daily user #623
Merged

This was referenced Aug 10, 2026
fix(push): a self-inflicted closed pipe must not mask the remote ingest error tracebloc/cli#472
Merged
fix(installer): enable GPU on Windows — CPU fallback + real GPU support (#616) #633
Merged

shujaatTracebloc mentioned this pull request Aug 12, 2026
fix(chart): stop init-writable-data skipping the chmod when the chown fails (#672) #689
Merged

Navigation Menu

Uh oh!

fix(installer): harden the Windows install→first-experiment path — checksum-driven downloads, reliable cluster-create, writable ingest & training volumes, local-chart support (#611) - #612

fix(installer): harden the Windows install→first-experiment path — checksum-driven downloads, reliable cluster-create, writable ingest & training volumes, local-chart support (#611)#612
shujaatTracebloc merged 11 commits into
developfrom
fix/609-checksum-driven-download

shujaatTracebloc commented Aug 5, 2026•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

saadqbal left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursorBot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

shujaatTracebloc commented Aug 5, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What & why

Fixes

1. Checksum-driven tool download (k3d / kubectl / helm) — install-k8s.ps1

2. Reliable cluster-create — install-k8s.ps1

3. Writable hostPath volumes for ingest and training/inference — chart (jobs-manager-deployment.yaml)

4. Windows local-chart support (TRACEBLOC_CHART_PATH) — install-k8s.ps1

Tests

Scope note

CI note

Uh oh!

Uh oh!

Uh oh!

Uh oh!

saadqbal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursorBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shujaatTracebloc commented Aug 5, 2026•
edited
Loading

1. Checksum-driven tool download (k3d / kubectl / helm) — `install-k8s.ps1`

2. Reliable cluster-create — `install-k8s.ps1`

3. Writable hostPath volumes for ingest and training/inference — chart (`jobs-manager-deployment.yaml`)

4. Windows local-chart support (`TRACEBLOC_CHART_PATH`) — `install-k8s.ps1`