Uh oh!
There was an error while loading. Please reload this page.
release-train: staging -> main - #644
Merged
Merged
Conversation
…ecksum-driven downloads, reliable cluster-create, writable ingest & training volumes, local-chart support (#611) (#612) * fix(installer): make the checksum drive the download retry so a truncated tool binary self-heals (#611) Field follow-up to #607/#608. On a Windows machine behind a filtering proxy, the k3d download kept failing at "System tool checksum verification failed" even with #608's multi-transport download — because #608 validated a download by SIZE FLOOR + magic bytes only. A binary truncated mid-transfer to somewhere between the 10 MB floor and the real 25.8 MB still passed (it's >10 MB and starts with 'MZ'), so the fallbacks never fired and it dead-ended at the separate, no-retry checksum step. (Proven on the box: a manual download produced the correct hash at 25,805,312 bytes, while the installer's copy failed the checksum in ~2s with no fallback.) Fix — the checksum is the authoritative completeness test: - Get-VerifiedDownload gains -Sha256: after a transport lands a size/magic-valid file, its SHA-256 must equal the expected hash or the transport is treated as failed and the NEXT one (curl.exe -> BITS) is tried. A truncated/altered copy now self-heals instead of dead-ending. - Get-VerifiedDownload gains -MustContain for the small checksum-list files, so a proxy error page lacking the expected asset line is retried too. - k3d / kubectl / helm now fetch their checksum FIRST (resiliently) and pass the extracted, 64-hex-validated hash as the download gate. helm gains checksum verification on the PS path for the first time (parity with the bash path). Tests: Pester source-guards for the -Sha256/-MustContain gates, the mismatch->retry path, and per-tool checksum-first wiring; full suite green (444). Manifest regenerated. Contributes to #578. Closes#611. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#611): content-gate the kubectl .sha256 fetch so a proxy page retries transports (Bugbot) The kubectl .sha256 is a bare 64-hex hash with no fixed substring, so it used -MinBytes 1 with no content gate -- a proxy error page satisfied the floor, the first transport 'succeeded', curl.exe/BITS never ran, and the later hex check aborted. Add -MatchPattern (a regex content gate) to Get-VerifiedDownload and use '[0-9a-fA-F]{64}' for the kubectl checksum fetch, matching how k3d/helm use -MustContain. Pester guards updated; manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): don't misread a successful k3d cluster-create as failed (#611) Field report (same Windows box, past the k3d download fix): Step 3 aborted with "Failed to create compute environment" even though k3d printed "Cluster 'tracebloc' created successfully!" with EMPTY stderr and the cluster was actually up. Cause: Wait-ProcessWithDeadline polled HasExited but never called WaitForExit(), and Start-Process -RedirectStandardOutput can leave $proc.ExitCode $null in that window -- so `$null -ne 0` misread an exit-0 success as a failure. Not machine-specific; a latent race any Windows user can hit. - Wait-ProcessWithDeadline now calls $Process.WaitForExit() before returning success, so the redirected streams drain and ExitCode is reliable for EVERY caller (cluster create, partial delete, tracked installs). - Cluster-create adds defense-in-depth: a still-null exit code falls back to k3d's own "created successfully" marker rather than failing a cluster that is up. - Pester source-guards for both. Windows-only change (install-k8s.ps1); Linux/mac paths untouched and their suites remain green. Contributes to #578. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#611): hash-anchor the checksum-list gates + check both k3d streams in the exit fallback (Bugbot) Two Bugbot findings on the checksum-driven download work: 1. (High) The checksum-LIST fetch gates were fail-open, so a proxy error page "succeeded" on the first transport and skipped the curl.exe/BITS retry — exactly the case #611 exists to survive. Helm's -MustContain substring (helm-<ver>-windows-<arch>.zip) also appears in the request URL a proxy page can echo; kubectl's -MatchPattern was unanchored so any page with a 64-hex run passed; k3d gated on the bare asset name. Fix: drop the weak -MustContain entirely and gate every checksum-list fetch on the hash STRUCTURE — k3d/helm require a 64-hex hash adjacent to the asset, kubectl requires the hash at the start of the body. A proxy/HTML error page can't satisfy that, so it retries transports as intended. 2. (Medium) The null-exit-code cluster-create fallback only scanned $k3dStdout, but k3d logs its "Cluster created successfully!" line via logrus to STDERR — so a real success could be misread as failure. Fix: check both $k3dStdout and $k3dStderr. Pester source-guards updated: kubectl gate is start-anchored, k3d/helm gates are hash-anchored, no -MustContain remains, and the fallback inspects both streams. Full suite green (447). Manifest regenerated. Contributes to #578. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#611): make /data/shared writable so dataset ingest works on hostPath installs `tb data ingest` failed at the copy step with `mkdir: can't create directory '/data/shared/.tracebloc-staging/': Permission denied`. On hostPath installs (the Windows/WSL2 + bare-metal default) kubelet does not apply fsGroup to hostPath volumes (kubernetes/kubernetes#138411), so /data/shared (client-pvc) is created root-owned and the non-root ingest-staging pod can't write to it. mysql-data has a privileged init-chown for exactly this reason; the shared data volume had none. - jobs-manager gains fsGroup: 1000 (CSI clusters apply it to the shared volume). - On hostPath, a privileged init-shared-data container (root, CHOWN+FOWNER only) chowns /data/shared to 1000:1000 and chmod 2777. World-writable, unlike mysql-data's single-UID chown, because the shared volume has multiple non-root writers whose UIDs this chart doesn't control -- jobs-manager, the training/ ingestor pods it spawns, and the CLI's ingest-staging pod. setgid keeps new files in GID 1000; the init is gated on hostPath (CSI relies on fsGroup). - helm-unittest: init present + world-writable on hostPath; absent (fsGroup kept) on CSI. Chart bumped 1.9.15 -> 1.9.16. Client-side companion to the installer fixes on this PR (requested to land here). Contributes to #578. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): support TRACEBLOC_CHART_PATH on Windows (local-chart parity with bash) The Windows installer could only ever install the PUBLISHED chart (helm repo), so a branch-only chart change (e.g. the #611 /data/shared fix, or #585's global.imageRegistry) was impossible to test from a Windows install. The bash installer already supports a local chart via TRACEBLOC_CHART_PATH (_resolve_chart_ref); this brings Windows to parity. - When $env:TRACEBLOC_CHART_PATH is set, install-k8s.ps1 installs from that local chart directory (validated) and skips `helm repo add`; otherwise it uses the published repo as before. Applied to both the fresh-install and adopt/reconcile helm upgrades. - Pester source-guards for the local-chart ref, the repo-add skip, and the not-a-directory error. Manifest regenerated. Full suite green (451). Enables a from-scratch Windows test of the branch chart. Contributes to #578. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): make /data/logs writable for training & inference pods (#611) hostPath ignores fsGroup (kubernetes/kubernetes#138411), so /data/logs was created root-owned and non-root training/inference pods hit `PermissionError [Errno 13]` creating their per-run log dir (`os.makedirs('/data/logs/<run>')`). The #611 init-container chowned /data/shared but not /data/logs — the same class of bug on the logs volume. Extend the init (renamed init-shared-data -> init-writable-data) to chown+chmod BOTH hostPath volumes and mount both. Training and inference pods share one spec (job.yaml), so this covers both; ingestion already covered by the /data/shared chmod. Bump chart 1.9.16 -> 1.9.17. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): address review — sticky bit on shared dirs + reset-then-reuse parity Two reviewer follow-ups on #612: - chart: chmod the writable hostPath dirs 3777 (was 2777) — add the sticky bit so one writer can't unlink/rename another writer's files in /data/shared // /data/logs (/tmp semantics). setgid is retained. Safe given the uid topology (dir owned by 1000; training pods run as 1000; the ingestor writes its own subtrees as a stable uid) and no cross-uid filesystem deletes exist in client-runtime. Chart 1.9.17 -> 1.9.18. - install-k8s.ps1: the adopt/reconcile helm upgrade now prefers --reset-then-reuse-values when `helm upgrade --help` advertises it (Helm >= 3.14), falling back to --reuse-values otherwise — so NEW chart defaults reach adopted Windows edges on auto-upgrade (bash parity with install-client-helm.sh). manifest.sha256 regenerated. Tests: helm-unittest updated (3777); new Pester test asserts the reset-then-reuse preference; full Pester suite 452 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): add CAP_FSETID so the init's setgid bit survives (Bugbot) init-writable-data chowns the shared/logs hostPath dirs to GID 1000 then chmods 3777. With caps dropped to CHOWN+FOWNER only, the kernel silently strips S_ISGID on the chmod — after the chown the dir's group no longer matches the process (fsgid 0), and a root process without CAP_FSETID can't set setgid on it — so the mount landed at 1777 and new files did NOT inherit GID 1000 as documented. Add FSETID to the cap set; setgid now sticks. helm-unittest asserts FSETID present. Chart 1.9.18 -> 1.9.19. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): drop risky fsGroup + make init per-volume best-effort (Bugbot) Two Bugbot findings on the writable-volume fix: - HIGH — remove `fsGroup: 1000` / `fsGroupChangePolicy` from jobs-manager. It is a no-op on hostPath (kubelet ignores fsGroup — the init does the work) and on CSI it only grants jobs-manager's OWN processes GID 1000 while its OnRootMismatch relabel flips the shared/logs volumes to group 1000 — stripping the group-0 access the spawned training pods (UID 1001 / OpenShift arbitrary UID, GID 0) and the host-UID ingestion pods rely on (docs/SECURITY.md §5.3). It never reaches those spawned writers, so it was all regression risk and no gain. Those pods keep their own documented posture; CSI is untouched (matches develop). - MEDIUM — the init now fixes each dir INDEPENDENTLY and best-effort: `for d in /data/shared /data/logs; do chown && chmod || echo <warn>; done`. A chown that can't complete (e.g. /data/shared on an NFS root_squash export) no longer aborts the chain and skips /data/logs — the other dir is still repaired and jobs-manager still starts; a truly unwritable mount surfaces as a clear error at the writer pod instead of wedging the edge in Init. helm-unittest updated (no fsGroup on either path; per-dir loop; CSI skips init). Chart 1.9.19 -> 1.9.20. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…r format guard (backend#723 PR-2) (#597) * feat(mysql): A2 engine split — 8.4 opt-in for fresh installs + datadir format guard (backend#723 PR-2) Chart: mysql-format-guard init container fails fast (with an actionable message) when the engine major and datadir format disagree — 8.4 over a 5.7-format datadir and 5.7 over an 8.x one are both refused before mysqld CrashLoops; the 8.0 transit hop and custom digest pins stand down. tracebloc.mysqlEngineMajor derives the expected engine (digest-wins, mirroring tracebloc.image); the 5.7 digest literal is CI-pinned to the values default. Default render changes by exactly the guard. Installer (A2, decision 2026-08-05): _resolve_mysql_engine picks the engine for the generated values — explicit TB_MYSQL_ENGINE wins; a previous 8.4 opt-in is sticky; any existing release or real datadir content pins 5.7; only a fresh arm64 install auto-selects 8.4 (native multi-arch instead of amd64 emulation). amd64 fresh installs stay 5.7 for now (soak first). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): fail CLOSED on an unlistable mysql datadir in the engine resolver (Bugbot) The auto branch treated a failed ls -A as an empty datadir — on arm64, --reuse-data commonly leaves a uid-999 mysql dir the host user cannot list, so the resolver opted the reuse into 8.4 and the format guard then (correctly) refused the 5.7 datadir: the reuse path never came up. An unlistable dir now counts as content (mirrors _leftover_data_dirs' fail- closed stance for the same ownership case), with a chmod-000 regression test. Rebased over #593/#527/#525 (manifest regenerated; my bats negations now carry the #527 '|| return 1' enforcement idiom). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(bats): harden the 11 new engine-resolver assertions per the #527 hygiene guard bats-hygiene's scanner requires every standalone bracket assertion in an @test body to end in '|| return 1'; the resolver tests added on this branch predated rebasing onto that guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(tests): restore the brace the #604 merge seam ate (last mirror test) git hoisted the shared closing brace out of the conflict region during the rebase onto #604; the file then died at parse (1 of 88 tests ran). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(chart): the format guard honors global.imageRegistry (Bugbot) Semantic rebase conflict with #604: every other image include gained the mirror dig while the guard (written pre-#604, merged clean textually) kept a hardcoded docker.io — on mirrored/air-gapped edges the always-on guard alone would ImagePullBackOff and block mysql on exactly the fleets #604 serves. Same dig expression now + a mirror re-home pin test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(chart): bump 1.9.20 -> 1.9.21, as this PR changes chart content My earlier merge resolution took develop's 1.9.20 verbatim, reasoning from the release train's rule: v1.9.20 is untagged, and the train's version_preflight only refuses when the version is ALREADY released, so one bump covers a whole release cycle. That reasoning is correct for the train and wrong for this repo. client/scripts/chart-version-guard.sh enforces a stricter rule for a repo-specific reason: chart content reaches installs only via a NEW chart version, because a Helm repo publishes on version change. An unbumped template/values edit therefore either reaches nobody or overwrites an already-published version. Both have happened here - the perIngestionTables block shipped dark in PR #472, and ingestor-0.2.0.tgz was overwritten 5x between 2026-05-20 and 2026-07-29. This PR changes client/templates/** and client/values.yaml, so it needs its own version rather than riding develop's. v1.9.21 is untagged. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…to verify' (client#631) On the curl|bash Linux path install.sh exports SSL_CERT_FILE from TRACEBLOC_CA_BUNDLE before launching install-k8s.sh, so wire_ca_trust always saw it pre-set, took the 'kept' branch, told the operator to verify their pre-set bundle (which the installer itself just set), and never reported the cosign/helm wiring. Treat a var already pointing at our CA (-ef) as wired; only a genuinely different pre-set takes 'kept'.
…#631) cluster.sh is a manifest-covered bootstrap file; its recorded sha256 was stale after the wire_ca_trust change, so the verified curl|bash installer would reject it.
…-message fix(installer): installer-set SSL_CERT_FILE shouldn't read as a pre-set bundle to verify (client#631)
…'t be fetched (#621) * fix(#556): fail closed when the macOS Docker Desktop DMG checksum can't be fetched The installer downloaded Docker Desktop's DMG and, if the .sha256sum fetch returned empty (a TLS-inspecting corporate proxy, a transient CDN error), logged "skipping verification" and went on to `hdiutil attach` + copy the DMG into /Applications under sudo — installing an UNVERIFIED image on a privileged path. That is a supply-chain fail-open, in contrast to the Windows k3d/kubectl downloads which fail closed. Retry the checksum fetch a few times, then ABORT with a clear, actionable message if it still can't be fetched, unless the operator explicitly opts out via TRACEBLOC_ALLOW_UNVERIFIED_DOCKER_DMG=1 (matching this file's existing TRACEBLOC_ALLOW_* override convention). The mismatch path already aborted and is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): regenerate scripts/manifest.sha256 for setup-macos.sh change (#556) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#556): verify macOS DMG via checksums.txt; warn-not-brick when unfetchable The floating Docker.dmg.sha256sum sibling 403s, so the prior fail-closed-on-empty would have aborted EVERY clean-network install. Switch the source to the co-located checksums.txt (returns 200; BSD format "<sha256> *Docker.dmg"), which genuinely verifies the exact floating DMG being downloaded and still fails closed on mismatch. When no checksum can be fetched (proxy/VPN stripping, CDN hiccup, layout change), emit a loud warning and proceed instead of bricking; operators wanting strict fail-closed can set TRACEBLOC_REQUIRE_DOCKER_DMG_CHECKSUM=1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(#343): stop stamping GPU_LIMITS on CPU-only installs Only render the GPU_LIMITS env var when a GPU is actually configured (a non-empty .Values.env.GPU_LIMITS). A CPU-only install writes GPU_LIMITS: "" and a chart-direct install may omit the key entirely; previously the template still emitted the var (or its "nvidia.com/gpu=1" default), surfacing a phantom GPU in the CLI resources view that the CLI had to normalize away client-side. Applied to both the api and pods-monitor containers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#343): gate GPU_REQUESTS on same condition as GPU_LIMITS (#617 bugbot) A chart-direct install that omits env.GPU_LIMITS was still emitting GPU_REQUESTS via its "nvidia.com/gpu=1" fallback, so CPU-only installs kept a phantom GPU and any chart-direct install could ask for a GPU request with no matching limit (rejected by the API server). Gate GPU_REQUESTS with the same `(default "" .Values.env.GPU_LIMITS)` condition in both the api and pods-monitor containers so a CPU-only install emits neither var and a GPU install emits both (request==limit). Also bump client/Chart.yaml version 1.9.15 -> 1.9.16 so the chart content change is publishable (chart-version-guard). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
release-train: develop -> staging
…stall (#618) * fix(#553): wait out the metrics-server APIService race before helm install On a freshly created k3d cluster, k3s registers its bundled metrics-server (and the v1beta1.metrics.k8s.io APIService) shortly AFTER the API server is ready. `k3d cluster create --wait` gates only on node/serverlb readiness, so on a slow WSL2/laptop `helm upgrade --install` could render the resource-monitor DaemonSet in that window and hit its `{{ fail }}` guard, aborting the entire first install. Add a bounded, best-effort `kubectl wait` for the metrics APIService before the helm install (TB_METRICS_WAIT_S, default 120s). If it never registers we fall through and let the chart's render-time guard produce its actionable error, so a genuinely missing metrics-server is still caught (issue's preferred option (a)). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(installer): refresh manifest.sha256 + address #618 review - metrics APIService probe passes --request-timeout=10s (non-watch kubectl rule) - outer spinner deadline derived from TB_METRICS_WAIT_S (+60s slack) so a configured wait >180s is no longer silently truncated - regenerate scripts/manifest.sha256 for the changed install-client-helm.sh Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#553): skip the metrics-server wait under test/no-kubectl so bats stays fast The unconditional _wait_for_metrics_apiservice poll loop sleeps 3s up to ${TB_METRICS_WAIT_S:-120}s. In the mocked install_client_helm bats tests kubectl isn't stubbed, and on the CI runner it isn't installed at all — the `kubectl get` just fails instantly, so the loop still burns its full deadline every test, blowing the 10-min job cap (Bugbot #618, CID 3728079502). Gate the wait exactly like the neighbouring network-y step _download_services_progress: no-op when TB_NO_SERVICE_PROGRESS is set (the bats suite sets it in setup()) or when kubectl is unavailable. Real installs set neither and always have kubectl, so the wait is unchanged for them. Regen manifest.sha256. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Removes this repo's caller for the public PII gate. Lukas's decision, 2026-08-06: retire the gate for now rather than leave it in place undeployed. WHY. The gate has never had a denylist. PII_DENYLIST does not exist as an org Actions secret and never has, so the workflow correctly failed closed on every PR - "an inactive guard must be visible as red, not invisible as green", as its own header puts it. Applied to a secret that was never provisioned, that produced a permanent red across all 7 public repos which blocked nothing: 24 open PRs carried it, and the whole org - this author included - had learned to filter it out. A check that is red on every PR is not fail-closed; it is off with extra steps, and it would be indistinguishable from noise on the day it went red for a real reason. WHAT LAPSES, stated plainly. The gate grepped PR title + body + every commit message against a private list of customer/partner names, on public repos only. It never read code, diffs or filenames, and detected no PII patterns - secrets in code remain gitleaks' job. So what lapses is the mechanical backstop for CLAUDE.md's rule that internal work and customer mentions never go in a public repo. That rule still stands and is now enforced by review alone. The reusable workflow and its selftest stay in tracebloc/.github, dormant with no callers. Restoring is three steps: set the org secret, restore this file, flip the inventory entry back to required. Checklist and the denylist-content caveat are on backend#1409. Not required on any branch, so this removes no merge gate.
#623) * fix(#558): write Docker Desktop autostart for a provisioned daily user Set-DailyUserProvisioning only wrote the HKCU Run key when the provisioned user was the account running the installer ($user -eq $env:USERNAME). For a provisioned DIFFERENT daily user (the hospital IT-installs-elevated case), their registry hive isn't loaded, so no autostart was written and the code relied on --always-run-service. On the WSL2 backend dockerd runs inside the docker-desktop distro that the Docker Desktop GUI boots, so without the GUI autostarting on that user's login the engine isn't up after a reboot and the k3d containers have no daemon to restart into — the client is down until someone opens Docker Desktop manually. For the provisioned different user, drop a "Docker Desktop.lnk" into their Startup folder (same launch-at-logon mechanism as the Run key, no hive load needed). When they have no profile yet, name the one-click GUI setting to flip after first sign-in. The current-user Run-key path is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(installer): refresh manifest.sha256 for changed install-k8s.ps1 (#558) scripts/install-k8s.ps1 changed in this branch; regenerate the signed integrity manifest via scripts/gen-manifest.sh so the "Static analysis" gate (gen-manifest.sh --check) passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#558): surface autostart failures instead of a silent green summary The cross-user Docker Desktop autostart path (Startup-folder shortcut for a provisioned DIFFERENT daily user) can fail on Startup-dir creation, the WScript.Shell COM object, or saving the .lnk. It was caught by the outer `catch { Log ... }`, which logged but added nothing to $did. With docker-users succeeding, the summary then printed a green "Configured for '$user'" with no autostart note -- so IT left the elevated window thinking #558 was handled while the daily user still had no autostart and the client is down after every reboot. Append a manual-step note to $did on failure, mirroring the .wslconfig catch in the same function, and name the one-click GUI remediation. Add source-guard Pester tests for the Startup-shortcut feature and the new failure-note path; refresh manifest.sha256. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(gpu): custom k3s-CUDA node image + publish workflow (#616) Foundational build artifact for GPU-enabled edges, split into its own PR so it can land on develop and be published via CI (workflow_dispatch only dispatches from the default branch). Inert on develop — nothing references it until the installer PR (#633) wires it in. The stock rancher/k3s image is Alpine with no NVIDIA runtime, so GPU pods can never schedule on it. This image rebuilds the SAME pinned k3s on an NVIDIA CUDA Ubuntu base, installs the NVIDIA Container Toolkit, configures containerd for the `nvidia` runtime, and bakes in the device plugin + `nvidia` RuntimeClass so the node advertises nvidia.com/gpu on first boot. Based on the official k3d CUDA recipe. - docker/k3s-cuda/Dockerfile, nvidia-device-plugin-daemonset.yaml, build.sh, README.md - .github/workflows/build-k3s-cuda.yaml (manual dispatch; push=true publishes to GHCR, which also validates the Dockerfile builds) Once published (public), the installer pulls it automatically at cluster-create — the end-user runs one command; no manual build or pull. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): resolve Bugbot findings on the k3s-CUDA build infra (#635) - workflow: pass GITHUB_TOKEN + actor via `env:` and reference quoted shell vars instead of interpolating context straight into the run line of a packages:write job (GHA injection hardening). - workflow: add `timeout-minutes: 60` so a stalled nvcr.io pull / wedged push fails fast instead of spinning to the 6h default (parity with sibling workflows). - Dockerfile: the NVIDIA keyring + apt-list curls now carry the TLS floor + bounded timeouts inline (`--tlsv1.2 --connect-timeout 30 --max-time 60`), the way scripts/install.sh spells them out (house rule; a Dockerfile can't source curl_secure()). - drift guard: the k3s pin was hardcoded in the Dockerfile ARG, build.sh, and the workflow input default with a comment CLAIMING the workflow enforced it — nothing did. Wire all three into check-facts.sh so they're verified == facts.env K8S_VERSION; a bump now fails CI (and `--write` restamps them) instead of silently deriving a GPU image tag that was never published (#547 class). check-facts.sh --check passes; shellcheck clean; workflow YAML valid. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(gpu): seed the k3s-CUDA consumers into the check-facts sandbox (#635) Adding the GPU image's k3s pin as check-facts consumers made check-facts.bats fail — the throwaway repo the tests build only seeds scripts/, so the new Dockerfile / build.sh / workflow consumers reported "file not found" and the --check/--write tests went red. Seed those three files into setup() (matching the spec) so the drift-guard tests exercise them too. Also proves the workflow extractor picks the k3s_tag default (v… tag) and leaves cuda_tag alone. Local: check-facts.bats 14/14, bats-hygiene 18/18. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(gpu): run the check-facts drift gate on GPU-image changes too (#635) The check-facts --check gate lives in installer-tests.yaml, whose triggers were path-filtered to scripts/**. A change touching only docker/k3s-cuda/* or the build workflow never ran the gate, so a K3S_TAG edit in the GPU image could drift from facts.env and merge green — defeating the drift rows just added (Bugbot). Add docker/k3s-cuda/** and the build workflow to both the push and pull_request path filters so the gate runs on those changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…#622) * fix(#557): preflight port 6550 on Windows so a conflict fails clearly New-K3dCluster binds the cluster API server to 127.0.0.1:6550, but Test-Preflight never checked whether 6550 was already in use. On a machine where a leftover/other k3d cluster or an unrelated service owns 6550, `k3d cluster create` failed and the installer surfaced k3d's raw stderr instead of a clear "port already in use" cause. Add Get-PfPortListening (Get-NetTCPConnection based; $null when undeterminable) and a port-6550 check to Test-Preflight. A port owned by THIS installer's own already-running cluster is fine (that run reuses it), so the check only hard-fails when the listener is NOT our cluster, with actionable guidance to find and stop the owner. Adds Pester coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#557): harden port-6550 preflight (bounded, fail-closed, running-gated) Address PR #622 review on the Windows port-6550 preflight: - Get-PfPortListening no longer fails open: -ErrorAction SilentlyContinue swallowed real CIM/access errors into the same empty result as a free port, so a busy port green-OK'd. Switch to -ErrorAction Stop and, in the catch, treat only Get-NetTCPConnection's ObjectNotFound (no listener) as "free"; every other error returns $null ("can't tell"). (Bugbot Med) - Port-ownership check now reuses Test-ClusterRunning instead of a bare `k3d cluster list -o json`: it wraps the call in the same ~15s job deadline, so a wedged Docker engine can't hang preflight (Bugbot High), and it gates on serversRunning >= 1, so a STOPPED leftover cluster no longer masks a foreign listener on 6550 (Bugbot Med). - Regenerate scripts/manifest.sha256 for the changed install-k8s.ps1. - Add Pester coverage: probe error -> $null, ObjectNotFound -> free, stopped-leftover + foreign listener -> fail, running-owned -> ok. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(#557): give the port-6550 running-cluster case a default Has mock After the rebase merged our port-6550 Pester tests with #612, Test-Preflight's network-reachability block calls Has for kubectl/helm/k3d. The "port 6550 in use by our running cluster -> ok" case passes ownership, so it now continues into that block, where a k3d-only -ParameterFilter left those later Has calls with no matching mock ("No mock for command 'Has' matched"). Use a plain default mock (k3d + tools present -> only always-critical hosts probed) so every Has call is covered and the case does not throw. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#557): tri-state port-6550 ownership so a slow k3d list can't hard-fail reuse (Bugbot 3728340365) The port-6550 ownership check decided ownership solely from Test-ClusterRunning, whose bounded `k3d cluster list` maps BOTH a timeout AND a parse failure to $false. The ownership block treated that $false as "foreign listener" and HARD-FAILED preflight with stop/delete hints, so a slow/wedged Docker on a normal re-run wrongly blocked the install and pointed the operator at their own cluster -- conflating "can't determine" with "definitely not ours". Make cluster run-state tri-state: - Get-ClusterRunStateFromList (pure) and Get-ClusterRunState (bounded) return 'running' | 'down' | 'unknown'. Get-ClusterRunState reports 'unknown' only when the list times out or its output is unparseable. - Test-ClusterRunningInList / Test-ClusterRunning become thin boolean wrappers (=='running'), preserving their existing contracts and callers. The ownership block now only hard-fails when CONFIDENT the listener is foreign ('down': enumerated and ours is absent/stopped, or no k3d installed). An 'unknown' read downgrades to a warning and proceeds, letting New-K3dCluster's start/repair path settle it instead of blocking the install. Tests: existing 3 cases updated to mock Get-ClusterRunState; added an indeterminate case (list timed out -> warns, does NOT hard-fail). Manifest regenerated. Pester not run locally (pwsh unavailable); validated by source analysis. Source-of-truth guards for Test-ClusterRunning still hold. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#557): classify an absent cluster as 'down', not 'unknown' (Bugbot High 3728714531) Get-ClusterRunState ran `k3d cluster list <name> -o json`, which fatals with empty stdout when the named cluster is absent. The classifier read that empty output as 'unknown' -> warn-and-proceed, so a genuinely-absent cluster with a FOREIGN listener on 6550 escaped the hard-fail -- exactly the "re-run after the first create failed on the busy port" case #557's preflight must catch. List ALL clusters instead (`k3d cluster list -o json`, no name) and look for $CLUSTER_NAME in the parsed array. A successful full list always emits at least `[]`, so absent-vs-error stays separable: - completed list without a running $CLUSTER_NAME (absent, stopped, or empty []) -> 'down' (confidently not ours -> hard-fail on a busy port) - only a TIMED-OUT or unparseable/failed list -> 'unknown' (warn-and-proceed) Tests: added a Get-ClusterRunState tri-state Describe -- pure classifier cases (absent/empty-list -> 'down'; empty/garbage -> 'unknown') plus bounded cases (timeout -> 'unknown'; completed-but-absent -> 'down'). Manifest regenerated. Pester not run locally (pwsh unavailable); validated by source analysis. Source-of-truth guards for Test-ClusterRunning still hold. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(#557): drop fragile job-cmdlet-mock tests for bounded Get-ClusterRunState (CI Pester) The two bounded-wrapper cases mocked Start-Job/Wait-JobWithProgress/Receive-Job, which is environment-fragile and fails under CI Pester. Coverage is retained via the pure Get-ClusterRunStateFromList tests + the Test-ClusterRunning source-guard (bounded-job regex). No product-code change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
#637) * fix(chart): image-refresh builds explicit in-cluster kubeconfig (#634) kubectl in the alpine/k8s image does not reliably auto-detect in-cluster config in the image-refresh Job pod: with no kubeconfig on disk it falls back to http://localhost:8080 and every API call fails, even though the SA token and KUBERNETES_SERVICE_* env are present. This made the refresh tick fail on every run fleet-wide (dev/stg/prod), surfacing only as the generic "cannot read deployment (NotFound, RBAC denial, or API error)" because the read gate swallowed kubectl's stderr. - Build an explicit in-cluster kubeconfig from the mounted ServiceAccount (server from KUBERNETES_SERVICE_*, ca + token from the projected SA volume) before any API call, so bare `kubectl` targets the real API server. Verified in a dev pod under the real pod constraints (readOnlyRootFilesystem, HOME=/home/kubectl, emptyDir mounts). - Surface the real API error on the fail path instead of >/dev/null 2>&1, so the next incident is one log line, not a live repro session. - Regression guards lock the kubeconfig preamble in place. - Bump chart 1.9.15 -> 1.9.16 (script is chart content; chart-version-guard). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(chart): address Bugbot — tokenFile off argv + IPv6-safe API host (#634) Two Bugbot findings on the in-cluster kubeconfig preamble: - SA bearer token was on the kubectl argv via `--token="$(cat ...)"` (readable through procfs). Use `kubectl config set users.*.tokenFile` — the kubeconfig stores only the PATH, no token value on argv or in the file, and kubectl re-reads the rotating projected token per call. - The server URL concatenated host:port without IPv6 brackets, an invalid URL when KUBERNETES_SERVICE_HOST is IPv6 (client-go uses net.JoinHostPort). Bracket the host for dual-stack / IPv6 clusters. Regression guards updated: assert bracketed api_host + tokenFile, and a notMatchRegex forbidding a regression back to `set-credentials --token`. Verified in a dev pod (readOnlyRootFilesystem, real SA): tokenFile path set, no token value embedded, bare kubectl rollout status + get both rc=0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
release-train: develop -> staging
* fix(#555): add activeDeadlineSeconds to the auto-upgrade Job autoUpgrade.timeout ("10m") bounds only `helm upgrade --wait`, not the earlier `helm repo add`/`helm repo update` HTTPS fetches. On a TCP black-hole (a corporate proxy that drops packets with no RST) those fetches can hang with no hard timeout; with concurrencyPolicy: Forbid the hung Job then makes every future scheduled tick "still active" and skipped, wedging auto-upgrade until a human kills the pod. Add activeDeadlineSeconds (new autoUpgrade.activeDeadlineSeconds knob, default 900s) to the Job spec as a hard wall-clock backstop, mirroring storage-assertions-check.yaml. Kept comfortably above the 600s helm timeout plus repo-update headroom so a slow-but-healthy upgrade is never killed mid-flight. Also declared in values.schema.json alongside the sibling knobs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#555): bump Chart.yaml version and nil-guard activeDeadlineSeconds Bump chart version 1.9.21 -> 1.9.22 so the chart-content change can be published (CI gate), and keep appVersion in lockstep with version ("1.9.22") per the repo convention — the previous divergence was the Bugbot finding. Default autoUpgrade.activeDeadlineSeconds to 900 in the CronJob template so a --reuse-values upgrade that omits the new key still renders a valid integer instead of an empty/invalid value. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#555): never let the Job deadline undercut a longer helm timeout Bugbot: the fixed default activeDeadlineSeconds (900s) could kill a healthy `helm upgrade --wait` when autoUpgrade.timeout was configured longer (e.g. 30m), firing before helm's own timeout. Render activeDeadlineSeconds as max(configured floor, parsed timeout + 300s repo-fetch headroom) via a new tracebloc.durationSeconds helper. The 300s mirrors the baked-in 900-600 default headroom, so the default case is a no-op (900) and the `| default 900` reuse-values nil-guard is preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(#561): run macOS install steps under bounded spinners Homebrew install, `brew install docker/colima`, `colima start`, and the `hdiutil` DMG install ran under spin_cmd, which has no deadline. A wedged brew (network stall), a hung colima start (stale VZ VM), or hdiutil on a bad DMG would hang the install indefinitely behind a spinning cursor with no timeout or kill. Move all five to spin_cmd_bounded with sensible deadlines (Homebrew 30m to allow a fresh-Mac Command Line Tools pull; the others 15m each), matching the create/helm paths that already bound themselves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(installer): regenerate manifest for setup-macos.sh change (#624) setup-macos.sh changed in this PR; scripts/manifest.sha256 was stale, failing the Static analysis --check gate. Regenerated via scripts/gen-manifest.sh; --check now passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ptops (#625) * fix(#562): soften API-wait and readiness timeouts for slow/proxied laptops F8 (_wait_for_api): the hard 60s cap (max=30 x sleep 2) false-failed a slow machine still loading images on its first kubectl even after `k3d --wait` returned. Replace with a seconds-based, env-tunable budget (TB_API_WAIT_S, default raised to 180s), and update the error to note a re-run is safe and how to extend the wait. F9 (READY_TIMEOUT): raise the default 300s -> 600s so several GB of images over a slow/proxied link doesn't leave a healthy-but-slow machine reported as CLIENT_STATE=starting / "not connected". Updated in all three synced consumers (scripts/spec/facts.env, summary.sh, install-k8s.ps1); check-facts.sh passes. Still overridable via the READY_TIMEOUT env var. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#562): align check-facts.bats fixtures with READY_TIMEOUT 600 spec (#625 Bugbot CID 3727520250) The spec (facts.env) raised READY_TIMEOUT 300 -> 600, but the bats fixtures still seeded the summary.sh / install-k8s.ps1 consumers at 300. That made the green-path --check test false-fail (a real drift the gate must NOT report) and left the drift test rewriting 300 -> 600, which no longer exercised drift once the spec was 600. Seed both consumers at 600 to match the spec, push the ps1 to a wrong value (900) in the drift test so the gate genuinely reports ReadyTimeout drift, and bump the --write test spec to 900 so it actually re-stamps both bash and PowerShell budgets. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…626) * fix(#563): back off image-refresh on a flapping-readiness deployment A deployment whose readiness oscillates (becomes Ready, then crashes after warmup right around the rollout-status timeout) fails the acting tick's `rollout status`. Under `set -e` the tick exits before the success annotation lands, so `recorded` never advances to the new digest and every later settled tick re-issues the identical `rollout restart` (~every schedule) — churning ReplicaSets on a genuinely flapping deployment. Record the ATTEMPT (target-digest signature + a consecutive-failure count) in a `tracebloc.io/refresh-attempt` annotation BEFORE the restart, so the counter survives the set -e exit. After imageRefresh.maxRefreshAttempts (default 3) in-a-row failures for the same target, skip the restart and surface the flapping deployment instead of re-restarting forever. The counter resets when the target digest changes or a rollout finally succeeds. New value + schema entry + env wiring; default installs stay byte-identical apart from the new env var. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#563): resume image-refresh once a flapping deployment settles + bound annotate (#626 review) Address PR #626 review: - Backoff was a dead end: after MAX_REFRESH_ATTEMPTS failures the tick exited before restarting forever, so a deployment that later became healthy/settled never got the reconciling rollout and stayed on the old image until a newer digest appeared. Now, while in back-off (which is only reached on a tick that already passed the settled guard), each settled observation advances a cooldown; after BACKOFF_RESUME_AFTER settled ticks the attempt counter resets and refresh retries. A cleared flap resumes and reaches `recorded == latest`; a still-flapping deployment is retried at most once per cooldown, never every tick. - Bound the new pre-restart `kubectl annotate` calls with --request-timeout=15s (repo rule for non-watch kubectl). - values.yaml comment updated to describe the resume-on-settle behaviour. - Chart.yaml version 1.9.15 -> 1.9.16 (chart content changed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#563): read the back-off attempt counter fail-closed (#626 review) The attempt-counter read used `get_annotation ... || true`, and get_annotation piped `kubectl | jq` with no pipefail. A failed kubectl/jq therefore collapsed to an empty value that the back-off logic read as "no prior attempts" (0) -- failing OPEN past the guard and re-restarting a possibly-flapping deployment (violates the no-fail-open-guards rule). get_annotation now captures kubectl separately and returns non-zero on either a kubectl or jq error, so a caller can distinguish a genuinely absent annotation (success + empty output -> 0 attempts) from a read error (non-zero return). The attempt read takes the SAFE branch on a read error: skip the restart this tick rather than churn the deployment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#563): resume grants one restart, not a burst; fix cooldown off-by-one (#626 review) Two back-off bugs in the image-refresh cooldown logic: 1. Cooldown resume restarted in bursts (CID 3728456918). On resume the counter was reset to 0, handing a still-flapping deployment a fresh budget of MAX_REFRESH_ATTEMPTS restarts across the next consecutive settled ticks -- a ReplicaSet-churn burst that broke the "at most once per cooldown" invariant. Now resume sets prev_count = MAX-1 so exactly ONE restart is issued; if it fails, the persisted count is MAX and the next settled tick re-enters back-off immediately. A healthy deployment still succeeds on that one attempt and clears the counter. 2. Cooldown off-by-one delayed resume (CID 3728456929). The branch compared settled_ticks (excluding the current tick) against BACKOFF_RESUME_AFTER, so the tick logging "Settled 4/4" still incremented and exited, and resume happened only the following tick (~15m late). Now it compares the count INCLUDING this settled tick, so 4/4 resumes on that tick, matching the log and the "after BACKOFF_RESUME_AFTER settled ticks" contract. Both fixes reuse the existing single-counter encoding; no new state added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#563): bound get_annotation read with --request-timeout=15s so a wedged API fails closed, not hangs (#626 Bugbot CID 3728757993) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#563): replace image-refresh back-off with a plain consecutive-failure counter (#626 Bugbot CID 3728806670) Simplify the over-engineered cooldown/auto-resume/signature machinery down to the minimal design #563 calls for. The single-integer annotation is no longer overloaded (signature + count + cooldown offset); it is now a plain count of consecutive failed refresh attempts that means one thing. Removed: - the cooldown counter, BACKOFF_RESUME_AFTER, cooldown_seen/settled-tick counting, and the resume-to-MAX-1 single-restart logic (auto-resume); - the signature built from annotate_args and its target-change reset. This is the fragile "partial resolve resets back-off" path (Bugbot CID 3728806670) -- deleting it eliminates the finding. Kept (the correct #563 core): - settled-guard short-circuit for unsettled deployments; - digest-change detection (refresh only when the resolved digest changed); - fail-closed annotation reads (skip the tick on a kubectl/jq read error, never treat an error as "0 attempts"); - --request-timeout=15s on every non-watch kubectl call. New semantics: increment the counter (persisted before the restart) when a rollout is issued but does not settle; reset to 0 ONLY on a genuinely successful settled rollout; after MAX_REFRESH_ATTEMPTS in a row, STOP restarting, WARN, and annotate tracebloc.io/refresh-flap-detected for a human / monitoring. No auto-resume -- a human clears the counter (or a rollout that finally settles resets it). Net -30 lines. helm lint + template render clean; embedded script passes sh -n / dash -n; tick-by-tick simulation confirms flap->stop+flag (no burst), healthy->reset, read-error->skip, new-digest->refresh, unsettled->short-circuit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#563): make the image-refresh success reset bounded + independent of the digest write (#626 Bugbot CID 3729110045) On a successful settled rollout the consecutive-failure counter (tracebloc.io/refresh-attempt) was cleared only by being batched into the final digest-record annotate, which had no --request-timeout. If that annotate hung or failed under `set -e`, the counter stayed elevated and the digest went unrecorded, so later ticks re-restarted, burned the budget, and could permanently trip the flap lockout while the CronJob stayed green. Clear the counter (and flap marker) in its own --request-timeout=15s annotate BEFORE the digest record, so the success reset is bounded and no longer contingent on the digest write succeeding. Also bound the digest-record annotate and the rollout restart with --request-timeout=15s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ds (#616) (#639) * fix(gpu): correct COPY --exclude placement in the k3s-CUDA Dockerfile (#616) The multi-stage rootfs copy failed with "cannot copy to non-directory: .../bin": --exclude was placed BEFORE the args (`COPY --from=k3s --exclude=/bin / /`), which the labs frontend didn't apply, so /bin (a real dir in the static-k3s image) was copied over Ubuntu's merged-/usr /bin symlink and collided. Match the official k3d recipe: `--exclude=/bin` AFTER the args, then copy /bin explicitly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): correct COPY --exclude (before args, relative path) + verify rootfs (Bugbot) (#639) A TRAILING `--exclude=/bin` is parsed as the COPY DESTINATION, so the k3s rootfs copied to /--exclude=/bin instead of / — the build passed but the image was BROKEN (no rootfs overlay). My earlier "fix" (moving --exclude after the args) hit exactly this. Correct form: --exclude BEFORE src/dest, with the path RELATIVE to the source (`bin`, not `/bin` — the absolute path never matched, which is why the very first build collided on /bin). Also make build.sh self-verifying: build + load, export the image filesystem, and FAIL if it contains a `--exclude` artifact or is missing /bin/k3s — so a mis-parse can never publish a broken image again. Only push after the rootfs verify passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): exclude all merged-/usr dirs so the k3s rootfs overlays cleanly (#639) The before-args `--exclude=bin` fixed /bin but the build then collided on /lib: Ubuntu 22.04 is merged-/usr, so /bin /sbin /lib /lib64 are all symlinks to /usr/*, while the Alpine k3s image ships them as real dirs. k3s is a static binary (no shared-lib deps), so exclude all four (keeping Ubuntu's glibc userland for nvidia-ctk) and overlay only k3s's static /bin into /usr/bin via the kept /bin symlink. build.sh rootfs verify relaxed to accept usr/bin/k3s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): exclude the full Ubuntu merged-/usr + /var symlink set (#639) /lib was not the last collision — /var/run (symlink to /run) collided next. Exclude every Ubuntu-22.04 top-level symlink dir preemptively (bin/sbin/lib/lib32/lib64/libx32 + var/run, var/lock); excluding a path k3s does not have is a harmless no-op. k3s is static, so keeping Ubuntu's glibc userland under those paths and overlaying only k3s's /bin is sufficient. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…xy Deployment (#641) * fix(#1143): add startup/liveness/readiness probes to requests-proxy The requests-proxy Deployment declared ports/securityContext/resources/env but no probes, so the pod was marked Ready the instant the container process started -- before gunicorn had bound :8888. `tb doctor`'s "Service Bus egress (requests-proxy) ready" check reads ReadyReplicas, so it showed a false green even when the proxy could not relay. Add tcpSocket startup/liveness/readiness probes on the gunicorn listen port (8888), mirroring mysql-deployment.yaml's three-probe structure. tcpSocket (not httpGet) because the proxy application lives in the separate client-runtime repo and exposes no chart-visible HTTP health path; a probe against an unknown path would 404 and wedge the pod, whereas a successful TCP accept is the honest "gunicorn is serving" signal the doctor check needs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(chart): bump version+appVersion to 1.9.26 (#1143) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(#1143): make requests-proxy probe comment honest about tcpSocket tcpSocket:8888 confirms the gunicorn master has bound the listen socket — a strict improvement over no probes that catches a container that never starts — but it does not fully close the false-green: the master binds before the worker finishes importing create_app() and keeps the socket bound while a worker crash-loops, so a proxy that can't relay can still read Ready. A real /healthz in client-runtime would be needed to close it fully. Probes themselves unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…t can't OOM-kill it (#642) * fix(#1144): pin jobs-manager memory requests==limits so mass restart can't OOM it On single-node cluster recovery (Docker/k3d restart -> every pod restarts at once and contends for node CPU + memory) the jobs-manager pod was killed with exit 137, both the `api` and `pods-monitor` containers, taking ~9m to recover. During that churn the client stops heartbeating and `tb` shows the alarming "couldn't confirm it's connected to tracebloc". Root cause: both containers ran Burstable QoS with requests.memory (512Mi/256Mi) well below limits (1Gi/512Mi). Under node memory pressure the kernel OOM-killer prefers the pod using more than its request, so jobs-manager was the victim. Fix (chart resources): pin requests.memory == limits.memory for both containers (api 1Gi, pods-monitor 512Mi) so each holds a Guaranteed memory reservation and is no longer the node's largest OOM victim -- the same requests==limits pattern mysql-deployment.yaml already documents. Also raise requests.cpu (api 100m->250m, pods-monitor 50m->100m) so a mass restart doesn't throttle their startup to the observed ~9m crawl. CPU limits and memory limits are unchanged. Updated both the values.yaml defaults and the template inline fallbacks (used on helm upgrade --reuse-values from a release predating these keys). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(chart): bump version+appVersion to 1.9.27 (#1144) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#1144): reword "Guaranteed QoS" to the accurate oom_score_adj mechanism (#642 review) The pod is Burstable (cpu request != limit); only memory is pinned. Pinning requests.memory==limits.memory lowers the container's oom_score_adj, making it a less-preferred node OOM victim — same pattern mysql uses (mysql also omits its cpu limit). Reworded the templates, values.yaml, and the test description. Also fixed the failing jobs_manager_test resource assertions: api requests.memory 512Mi -> 1Gi, pods-monitor requests.memory 256Mi -> 512Mi. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
LukasWodka
commented
Aug 10, 2026
ContributorAuthor
bugbot run |
Uh oh!
There was an error while loading. Please reload this page.
#643) * sec(#1528): provision tb_meta + tb_ingest service-account Secrets (S1) Chart side of backend#1528 S1 (RFC-0003 D10 close-out), paired with tracebloc/client-runtime's minting PR. jobs-manager mints two dedicated, single-database MySQL identities to replace the root-equivalent edgeuser; this wires the Secrets and env they read. - values: serviceDbAccounts (default false) + optional tbMetaPassword / tbIngestPassword operator pins, mirroring perExperimentDbCreds / credmgrPassword. Added to values.schema.json. - secrets.yaml: TB_META_PASSWORD / TB_INGEST_PASSWORD via the same generate-once, upgrade-stable 3-tier resolution as TB_CREDMGR_PASSWORD (operator pin -> existing Secret value -> randAlphaNum), alphanumeric-pin validation, emitted only when serviceDbAccounts is on. - jobs-manager-deployment: SERVICE_DB_ACCOUNTS=1 + TB_META_USER/TB_INGEST_USER + the two secretKeyRef passwords, rendered only when the flag is on. Gated so a default install is byte-identical (verified: `helm template` with the flag off renders none of the new keys). helm-unittest coverage added to secrets_test.yaml and jobs_manager_test.yaml (49 assertions, green). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * sec(#1528): bump chart version 1.9.27 -> 1.9.28 (chart-version-guard) templates/values/schema changed, so the chart-version guard requires a version: bump — a Helm repo only publishes a new version, and an unbumped edit ships dark or overwrites a published tarball. Additive feature => patch bump. appVersion unchanged (wiring only, no app image change). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#1528): bump appVersion to 1.9.28 in lockstep with version (Bugbot) .cursor/BUGBOT.md flags a Chart.yaml version bump without the matching appVersion — app.kubernetes.io/version derives from appVersion, so leaving it at 1.9.27 would tag upgraded resources as the previous release. Every prior bump moved the two in lockstep; this restores that. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…#1492) (#645) * ci: arm the action-pin gate — findings fail, not just report (backend#1492) The pinning work is done: 415 mutable action refs across the fleet on 2026-08-04, zero on every develop branch today (#1490, #1491 closed with that scan). This makes it stay done. action-pins has been running here since it reached main, but in soft-fail: it reports a violation and lets the PR merge anyway. That is the state hand-pinning already proved insufficient. While #1446 was open to pin one action, #1449 added a SECOND unpinned call site of the SAME action in a non-overlapping hunk of the same file. Both merged cleanly, git reported no conflict because there was none, and it was caught by a human reading the diff rather than by any gate. That incident is why #1492 exists. Arming imports no backlog. Measured immediately before this change, across all 15 develop branches carrying this caller: ZERO pin violations. So this flips a check that is already green into one that stays green. What it enforces, per D10: third-party and actions/* refs pinned to a full 40-character commit SHA with a trailing exact-version comment; tracebloc/* refs pinned to @main (Q3). A floating tag can be repointed by its owner at any commit, which is the supply-chain risk the whole class exists to remove. The .github repo armed this first and for a specific reason: it publishes these workflows to every other repo, so its own refs are inherited fleet-wide (.github#178, backend#1603). * fix(ci): put action-pins in the with: block, not workflow_dispatch inputs My first commit anchored on the bare key `all-files:`, which matched its FIRST occurrence in the file - the workflow_dispatch INPUT DEFINITION, not the code-quality call's `with:` map. So `action-pins: true` landed as an invalid input declaration (inputs need description/type), the workflow failed to load, and code-quality did not run at all on this PR. Measured: 10 of 15 callers hit this; the 5 that did not are the ones whose first `all-files:` happens to be in `with:`. Now anchored on the `with:` that follows the code-quality `uses:` line, so the insertion point is structural rather than a name that appears twice. Validated by parsing the result: action-pins present in jobs.quality.with, absent from workflow_dispatch.inputs. Also restores the trailing newline the first commit stripped.
release-train: develop -> staging
* docs(claude): sync org-standards block (backend#1602) Managed sync from tracebloc/.github/org-standards.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(claude): sync org-standards block (backend#1602) Managed sync from tracebloc/.github/org-standards.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(claude): sync org-standards block (backend#1602) Managed sync from tracebloc/.github/org-standards.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Bugbot #644 (High), recurrence of #629. The checksum fetch picked field 1 of any line mentioning "Docker.dmg". A TLS-inspecting proxy that returns an HTML error body mentioning that filename yields non-hash text, which was non-empty, so the code entered the fail-closed verify path, failed the compare, and hard-aborted an otherwise-fine install as "corrupted or tampered". Add the 64-hex structure check the PowerShell tool downloads already have: - awk now requires field 1 to be a 64-hex SHA-256 before capturing it - a belt-and-suspenders guard blanks any non-64-hex value, so garbage takes the intended warn/unreachable path instead of a fail-closed mismatch abort Real corruption/tampering still fails closed: a valid 64-hex hash that doesn't match the DMG aborts exactly as before (#556). Regenerated scripts/manifest.sha256 (setup-macos.sh is manifest-covered, R8). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
release-train: develop -> staging
* chore(devex): Makefile with a uniform 'make check' (backend#1606) Today every repo in the org has a different incantation for "run your tests": `python manage.py test` here, `yarn test:coverage` there, `make ci` in cli, `pytest tests/ -m "not slow"` somewhere else. That makes "run your tests before you push" a rule you can only obey if you already know the repo — which means it is not really a rule, it is tribal knowledge with a rule's wording. The people most likely to break it are exactly the people least likely to know the incantation. backend#1606 fixes that by giving every active repo the same three targets: make check lint + fast tests. Budget: UNDER 60 SECONDS. make check-all everything CI runs, minus the CI-only heavy suites. make setup install what those targets need. The 60-second budget is not decoration. It is the property that decides whether anyone runs the thing: a `make check` that takes ten minutes is a rule people learn to skip, and skipping one rule teaches skipping others. So the split between `check` and `check-all` is drawn on measured wall-clock time, not on taste. The Makefile is a THIN WRAPPER. Every command in it was lifted from the workflow that already runs it. It adds no tool, no config, and no rule, and it changes no CI workflow — making CI call `make check` is a separate, later wave (decision 2 on backend#1606), deliberately kept out of this PR so that a Makefile bug cannot redden the pipeline. No pre-commit or pre-push hook is installed here either; that is step 4. In this repo ------------ `make check` — MEASURED 5.3 s: bash -n on every shell script, shellcheck at error severity over the file set installer-tests.yaml lints, the three single-source drift guards (gen-manifest --check, check-facts --check, check-style), and helm lint --strict for all four platform values files plus the ingestor subchart. HONEST GAP — this is the one repo in the wave where I could not split fast from slow without lying. The bats suite IS this repo's unit suite (868 tests) and it takes ~2 minutes serially on macOS. It does not parallelise without GNU parallel, which is not on every dev machine, and the per-file timings show no natural fast tier: the slowest file (common.bats, 37 s) is also the most load-bearing, so any subset would be arbitrary and would rot. So bats sits in `check-all`, and `make check` here is lint-only. A `check` that quietly took two minutes would be a `check` nobody runs, which is worse. Splitting the bats suite is real work and belongs in its own ticket. `make check-all` adds bats, the 4-platform helm template render (with kubeconform if you have it), and helm unittest (with the plugin install line if you do not). CI-only by name: the 9-distro prereq matrix, e2e-cluster (k3d), e2e-proxy (squid), path-persist, Pester, windows-e2e, e2e-journey, and the three k3d seal/upgrade e2e jobs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(devex): the shell-syntax check no longer fails open on dash (Bugbot, backend#1606) Bugbot on #630, medium severity, and this is the best kind of finding: the check was not merely wrong, it was silently reporting success. Make runs recipes under `/bin/sh`. On Debian and Ubuntu that is dash, and dash rejects `read -d` outright ("Illegal option -d", reproduced locally). The `while IFS= read -r -d '' f` loop body would therefore never execute, the pipeline would still exit 0, and `make check` would print "all shell scripts parse" having parsed nothing at all. GitHub Actions runs its steps under bash, so CI never showed the problem — and `make setup` steers Debian/Ubuntu users straight into it. Replaced with `find -print0 | xargs -0 -n1 bash -n`: POSIX, NUL-safe for paths with spaces, and xargs propagates a child failure as a non-zero exit, so a real parse error now actually reds the check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
LukasWodka
commented
Aug 10, 2026
ContributorAuthor
bugbot run |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…rt (#616) (#633) * fix(installer): Layer 1 — never request a GPU the cluster can't provide (#616) On Windows a machine WITH a working NVIDIA GPU still ran CPU-only: the installer requested nvidia.com/gpu for every job whenever it merely DETECTED a GPU + driver, but the GPU is only provisioned in the cluster when K3D_GPU_FLAG is set (the WSL NVIDIA-toolkit step fully succeeded). When that step didn't complete, jobs got a GPU request the node couldn't satisfy -> Pending "Insufficient nvidia.com/gpu" until the SINGLE_NODE fallback downgraded them to CPU. GPU wasted, silently. Layer 1 (safe, always-correct; CPU fallback preserved): - Gate GPU_REQUESTS/GPU_LIMITS on K3D_GPU_FLAG (the SAME condition that provisions the GPU), not on mere detection. Empty gpuVal => no GPU request => training runs on CPU. The SINGLE_NODE downgrade stays as a belt-and-suspenders net. - Make the skip LOUD: capture a specific $GPU_SKIP_REASON at each toolkit early-return (WSL not responding / Ubuntu needs first-run setup / toolkit apt blocked or timed out / verify failed), surface it in the install summary ("CPU (GPU detected but not enabled: <reason>)") and warn during install. - Gate the doctor's GPU-test hint on K3D_GPU_FLAG too (don't suggest a test that can't run). Layer 2 (actually enabling the GPU in k3d-on-Windows) follows in this PR. Tests: 4 new Pester tests (values carry no GPU request when not enabled / carry it when enabled; summary shows CPU+reason vs NVIDIA GPU). Full suite 456 pass. manifest.sha256 regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): Layer 2a — custom k3s-CUDA node image for GPU edges (#616) The stock rancher/k3s image is Alpine-based with no NVIDIA container runtime, so GPU pods can never schedule on it. Add a drop-in replacement k3s node image that rebuilds the SAME pinned k3s (K3S_TAG == installer K8S_VERSION) on an NVIDIA CUDA Ubuntu base, installs the NVIDIA Container Toolkit, configures containerd for the `nvidia` runtime, and bakes in the device plugin + `nvidia` RuntimeClass so the node advertises nvidia.com/gpu on first boot. Based on the official k3d CUDA recipe. - docker/k3s-cuda/Dockerfile (multi-stage k3s + CUDA base) - docker/k3s-cuda/nvidia-device-plugin-daemonset.yaml (device plugin + RuntimeClass, pinned v0.14.5) - docker/k3s-cuda/build.sh (build/push, tag encodes k3s + CUDA versions) - docker/k3s-cuda/README.md - .github/workflows/build-k3s-cuda.yaml (manual dispatch; push=true publishes to GHCR) Image tag encodes both the k3s pin and CUDA base so a new k8s pin can't silently reuse a stale GPU image. The installer wiring to actually USE this image (cluster --image swap, GPU capability probe, RUNTIME_CLASS_NAME=nvidia) lands next in this PR; CPU fallback (Layer 1) always remains. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): Layer 2b — wire the installer to the custom k3s-CUDA image (#616) Make a Windows NVIDIA GPU genuinely usable by training pods, with CPU fallback whenever it can't be wired up. - Authoritative GPU gate: add Confirm-DockerGpu, which runs `docker run --rm --gpus all nvidia/cuda:<tag> nvidia-smi`. Docker Desktop uses its OWN WSL distro, so the toolkit-in-Ubuntu step was never a reliable signal; actually running a GPU container is. Enable GPU iff the probe passes -> we never create a `--gpus` cluster that would fail, and DD-GPU works even without the user's Ubuntu toolkit. Probe failure => K3D_GPU_FLAG cleared + a clear reason => CPU (Layer 1). - Cluster create: use the custom k3s-CUDA image ($K3S_CUDA_IMAGE, env-overridable via TRACEBLOC_K3S_CUDA_IMAGE, default ghcr.io/tracebloc/k3s-cuda:<K8S_VERSION>-cuda-<base>) when GPU is enabled; stock rancher/k3s otherwise. Same pinned k3s either way. - Set RUNTIME_CLASS_NAME=nvidia in values when GPU is enabled, so jobs-manager threads runtimeClassName: nvidia into every spawned pod (the RuntimeClass is baked into the CUDA image). Empty otherwise. Tests: Confirm-DockerGpu behavioral tests (pass/fail/no-GPU) + source guards for the image swap, probe gating, and RuntimeClass; existing GPU values tests extended for RUNTIME_CLASS_NAME. Full Pester suite 464 pass. manifest.sha256 regenerated. NOTE: end-to-end GPU execution needs validation on a real GPU Windows box (build + publish the image via the build-k3s-cuda workflow, then re-install). CPU path is unaffected and fully covered. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): make the GPU node image mirror-aware (one-command, air-gap) (#616) The one installer command must do everything with no external steps — the user never builds or pulls the GPU image by hand; the installer pulls it automatically at cluster-create. Re-home the default k3s-CUDA image onto TRACEBLOC_IMAGE_REGISTRY when a private mirror is configured (#585), so the single command also works on a restricted/air-gapped network, same as every other image. Explicit TRACEBLOC_K3S_CUDA_IMAGE still overrides. Source guard updated. Pester 464 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(installer): move k3s-CUDA image build infra to its own PR (#635) The Dockerfile + build workflow now live in #635 (targeting develop) so the image can be published via CI before this installer PR merges. #633 keeps only the installer wiring that CONSUMES the published image; nothing here changes. After #635 merges + the image is published, this branch rebases on develop to pick the build infra back up transitively. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): pull the private GPU node image with creds — keep it private (#616) We keep ghcr.io/tracebloc/k3s-cuda PRIVATE rather than public. So the installer must authenticate to pull it — the end user still runs ONE command; the creds come from env, not a separate docker login. - Confirm-GpuImagePullable: derives the registry host from $K3S_CUDA_IMAGE, `docker login`s with TRACEBLOC_REGISTRY_USERNAME/PASSWORD (the same vars the mirror uses, #585) when set, then `docker pull`s the image — which both VERIFIES access and PRE-LOADS it so k3d cluster-create reuses the local copy (no second pull). - The GPU gate now requires BOTH the passthrough probe AND a pullable image: `if ((Confirm-DockerGpu) -and (Confirm-GpuImagePullable))`. Either failure => clean CPU fallback with a specific reason (private-registry hint names the exact env vars to set; cred-set-but-failed says to check read access) instead of a cluster-create that dies pulling an unauthorized image. Tests: 4 Confirm-GpuImagePullable cases (login+pull ok / cred fail / no-cred hint / gate source-guard); updated the stale probe-gate guard. Full Pester 468 pass. manifest regen. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): bound the GPU docker calls with a timeout (Bugbot) (#616) The new `docker run` (GPU probe), `docker login`, and `docker pull` calls were unbounded — a wedged Docker daemon, registry, or proxy could hang Step 2 forever instead of falling back to CPU (installer external-call timeout rule). Route all three through a new Invoke-DockerCli helper that runs docker in a bounded background job (Wait-JobWithProgress deadline); on timeout it kills the job and returns Code=124 so callers fall back to CPU cleanly. Timeouts: probe 180s, login 60s, pull 900s (the GB CUDA image on a slow/proxied network — generous but finite). Login stdin (the token) is passed in-memory via the arg hashtable, never to disk, argv, or logs. Tests: rewired Confirm-DockerGpu / Confirm-GpuImagePullable tests to mock Invoke-DockerCli (regular function — mockable, unlike a job'd docker), added timeout-path (Code 124 -> CPU) cases + a source guard that every GPU docker call is bounded. Full Pester 482 pass. manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): drift-guard the CUDA tag via facts.env (Bugbot) (#616) CUDA_BASE_TAG (the CUDA base that derives the pulled GPU image tag) was hardcoded in four places — install-k8s.ps1, docker/k3s-cuda/Dockerfile ARG, build.sh, and the build-k3s-cuda workflow default — outside facts.env / check-facts, so a CUDA bump in one spot could leave the installer pulling a GPU image tag that was never built (#547-class drift, same as the k3s pin). Add CUDA_TAG=12.4.1-base-ubuntu22.04 to facts.env and wire all four consumers into check-facts.sh (verified == facts.env; --write restamps them). The workflow extractor distinguishes the cuda_tag default (…-ubuntu…) from the k3s_tag default (vX…k3s). check-facts.bats fixture seeds the two new consumers. check-facts.sh --check passes (15 consumers), check-facts.bats 14/14, shellcheck clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): kill docker on timeout — no orphaned native process (Bugbot) (#616) Invoke-DockerCli used Start-Job; Stop-Job stops the PS job but can orphan the native docker.exe it spawned, so a timed-out run/login/pull could keep going after the installer fell back to CPU. Run docker as a direct child Process instead and Kill() it on timeout, so the deadline actually terminates docker. 5.1-safe (.NET Framework Process API); args have no spaces (flags + image tags) so a plain join is safe; login stdin is written in-memory, never to disk/argv/logs. Callers are unchanged (still mock Invoke-DockerCli); updated the source guard to assert WaitForExit(timeout) + Kill(). Full Pester 482 pass. manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): accurate GPU-skip reasons + drift-check the GPU image (Bugbot) (#633) Two Bugbot findings: - GPU_SKIP_REASON now reflects the ACTUAL failure. A probe/pull timeout (Code 124) reports "timed out", not a GPU-unavailable or credentials error; a real GPU passthrough failure reports the Docker Desktop hint, not stale Ubuntu guidance. - Test-K3sVersionDrift now recognizes the GPU node image (…/k3s-cuda:<k3s>-cuda-<base>), extracting the k3s pin from before "-cuda-", so a GPU cluster no longer silently escapes the version-drift check after a pin bump (it only parsed rancher/k3s: before). Tests: reason assertions on the timeout/non-zero paths + a drift source guard for the GPU image. Full Pester 483 pass. manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): single-node GPU cluster so one card isn't double-counted (Bugbot) (#616) k3d's --gpus=all exposes the SAME host GPU to every node container, and the baked-in device-plugin DaemonSet registers it once per node. A default server+agent cluster therefore advertises nvidia.com/gpu=1 on BOTH nodes -- 2 allocatable for 1 physical card -- and Kubernetes can schedule two jobs onto the same device. Extra k3d nodes live on the same Docker host and all see the same card, so multi-node can never add real GPUs; it only double-counts. When the GPU is enabled, collapse to a single node (agents=0) so the card is advertised exactly once. An explicit user AGENTS is overridden loudly, not silently. CPU installs are unchanged (topology only shifts inside the K3D_GPU_FLAG="--gpus=all" branch). +1 source-guard test; full Pester 484 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): reused CPU cluster can't silently get GPU values (Bugbot) (#616) The GPU gate enables --gpus=all + GPU chart values in Step 2, before New-K3dCluster runs. On a re-install that REUSES an existing cluster (rather than recreating it), a cluster first built in CPU mode keeps its stock rancher/k3s node -- no NVIDIA runtime, advertises 0 GPUs, no `nvidia` RuntimeClass -- and k3s's node image is fixed at create time. Writing GPU values against it would strand every experiment Pending: exactly the #616 failure this PR removes. Reconcile in the reuse path: Confirm-ReusedClusterGpuCapable inspects the reused server node's image (bounded docker inspect, mirrors Test-K3sVersionDrift) and, when GPU was requested but the node isn't the custom k3s-CUDA image, disables GPU for this run (CPU fallback stays safe) and tells the user to recreate the cluster to enable GPU. The healthy fast-path already exits before the GPU gate, so it can't wrongly enable GPU. Pure Test-NodeImageGpuCapable classifier is unit-tested (stock/CUDA/mirror/ empty); the bounded inspect + clear-flag + call-site are source-guarded. Full Pester 492 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): build the GPU node image locally so GPU installs need no login (#616) Restores the one-command promise for GPU. The GPU node image was published private, so enabling GPU forced the user to supply a registry token -- exactly the friction #612 (CPU) never had, because every image it pulls is public. Default GPU path is now a LOCAL build: on an NVIDIA machine the installer runs `docker build` from PUBLIC bases only (rancher/k3s + NVIDIA's public nvcr.io CUDA base + the public NVIDIA container toolkit), tags it locally, and creates the cluster with that image -- no registry login, no private package, one command. Build-GpuNodeImage is idempotent (reuses an already-built image), bounded with a visible progress bar (Wait-ProcessWithDeadline), sanity-checks that the built image runs k3s, and on any failure falls back to CPU with a clear reason. The Dockerfile + device-plugin manifest are embedded (self-contained verified artifact) and a Pester drift guard keeps them byte-identical to docker/k3s-cuda/*. An explicit prebuilt image / mirror (TRACEBLOC_K3S_CUDA_IMAGE or TRACEBLOC_IMAGE_REGISTRY, i.e. air-gap tenants) still PULLS instead of building. CPU is untouched: the whole block is behind the NVIDIA gate + the docker-run probe, so a CPU machine builds nothing and gets stock rancher/k3s exactly as before. Windows-installer only; staging/prod (Linux + chart) are not affected. Full Pester 499 pass; check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): 5 GPU edge cases from Bugbot (latest, reuse, preflight, fallback) (#616) Round-3 Bugbot findings on the GPU installer, all CPU-safe: - Healthy reinstalls skip the GPU fix (Med): the completed+healthy fast path exited before any GPU reconciliation, so a cluster whose values request GPU on a CPU-only node kept stranding jobs while looking healthy. Add Test-HealthyClusterGpuConsistent to the fast path -- warns + recreate remedy. - K8S_VERSION=latest bypasses the CUDA image (Med): with latest, cluster-create adds no --image (stock node) but GPU stayed enabled -> stranded jobs. Refuse GPU for latest/empty K8S_VERSION at the gate (CPU fallback + reason). - Custom GPU image overrides fail reuse (Med): the reuse check only matched `k3s-cuda:`, downgrading a renamed/digest mirror image. Test-NodeImageGpuCapable now also accepts an exact match against the configured $K3S_CUDA_IMAGE. - Optional GPU build could abort the install (Med): Build-GpuNodeImage created the build-context dir OUTSIDE its try, so a temp/AV/disk error hit the fatal trap. Moved dir creation inside try + added a catch -> CPU fallback. - nvcr.io missing from preflight (Low): Find-Gpu now runs before Test-Preflight, which probes nvcr.io + nvidia.github.io when an NVIDIA GPU is present and we'll build. SOFT (warn, not hard-fail) since GPU degrades to CPU. +10 tests; full Pester 509 pass; check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): base64-embed the GPU build inputs so host lints don't trip (CI green) The raw here-string embed of docker/k3s-cuda/* broke two host-side guards that scan install-k8s.ps1: - check-style.sh flagged `apt-get install ... curl` (curl as a PACKAGE name) as a bare-curl call. - install.Tests.ps1's "no non-ASCII in string literals" test flagged the em-dashes in the embedded Dockerfile/manifest comments (an existing test, not to be rewritten). Embed the two files as base64 instead (pure ASCII, no `curl` token), decode them to the build context at build time. Same self-contained + drift-guarded property (the Pester drift test now decodes and compares to docker/k3s-cuda/*), with nothing for the container-vs-host lints to catch. No behaviour change to the build itself. Ran the FULL scripts/tests dir this time (both install.Tests.ps1 and install-k8s.Tests.ps1): 541 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): mirror-aware GPU probe + preflight the configured GPU registry (Bugbot) (#616) Two air-gap/mirror findings: - Mirror GPU path still hit Docker Hub (Med): the GPU passthrough probe pulled nvidia/cuda from Docker Hub regardless of the mirror. Add $CUDA_PROBE_IMAGE, re-homed onto TRACEBLOC_IMAGE_REGISTRY when set, and use it in Confirm-DockerGpu -- so a mirrored/air-gapped GPU install doesn't fall back to CPU on a blocked Docker Hub. (The device-plugin image baked into a PREBUILT node image is the mirror operator's build-time concern; the installer only pulls that image.) - Custom GPU registry bypassed preflight (Low): the new nvcr.io probes were added only on the build path, but the pull path contacts the configured registry. Add an else-branch that probes the $K3S_CUDA_IMAGE registry host (when it's a real host, not a bare Docker Hub repo) so an unreachable custom/mirror registry is surfaced at preflight, not at pull time. All GPU probes stay SOFT (warn -> CPU fallback), never hard-failing a CPU-capable install. Full scripts/tests: 543 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): regenerate embedded GPU build inputs from the merged Dockerfile (#616) Merged develop, which now carries #639's k3s-CUDA Dockerfile fix. PR CI builds the branch MERGED with develop, so the drift test compared the embedded base64 (generated from #633's older Dockerfile) against develop's newer one and failed. Regenerated the base64 embed from the current docker/k3s-cuda/Dockerfile + device-plugin manifest so the embed matches the merge tree. Full scripts/tests: 543 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): GPU registry login before the probe + correct Docker Hub host (Bugbot) (#616) Two mirror/auth findings from the mirror-aware probe change: - Private mirror probe ran before login (Med): with an authenticated TRACEBLOC_IMAGE_REGISTRY, Confirm-DockerGpu pulled the mirror-hosted CUDA probe image BEFORE Confirm-GpuImagePullable did `docker login`, so a credentialed install was rejected and fell back to CPU. Extracted Connect-GpuRegistry and call it in the gate BEFORE the probe (and reuse it inside Confirm-GpuImagePullable). - Docker Hub overrides logged into the wrong host (Low): for a bare `owner/image` ref, the login host was `owner` instead of Docker Hub. Added the pure Get-RegistryHost: a segment is a registry only if it has a '.'/':' or is 'localhost'; otherwise login targets docker.io. Full scripts/tests: 550 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): reconcile GPU on adopted-reuse + re-verify a reused GPU image (Bugbot) (#616) - Adopted releases retained stale GPU requests (Med): the adopted-reuse path uses helm --reuse-values, so an older release's GPU_REQUESTS/GPU_LIMITS survived even after cluster reconciliation cleared K3D_GPU_FLAG -> stranded jobs. Moved the GPU value decision before the adopted/fresh split and pass the three GPU env keys as --set-string on the adopted upgrade, so both paths reconcile GPU to this run. - Failed GPU images were reused unchecked (Med): Build-GpuNodeImage treated any existing tag as ready. A build that completed but failed the k3s sanity check left a broken image behind that the next run would reuse. Extracted Test-GpuImageRunsK3s and run it on the reuse path too; a broken existing image is rebuilt, not reused. Full scripts/tests: 554 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): node is authoritative for GPU + login to all GPU registries (Bugbot) (#616) - Mirrored installs could keep GPU on with a dead device plugin (Med): the device-plugin image is baked as nvcr.io/... and on a blocked mirror/air-gap network the DaemonSet never becomes ready, so the node advertises 0 GPUs while GPU requests stayed active -> stranded jobs. Confirm-GpuNode now makes the NODE authoritative: after the allocatable-GPU wait, a 0 count clears K3D_GPU_FLAG (before Install-ClientHelm writes values) -> CPU fallback with a clear reason. - Combined registry overrides logged into the wrong host (Med): with TRACEBLOC_K3S_CUDA_IMAGE and TRACEBLOC_IMAGE_REGISTRY on different hosts, Connect-GpuRegistry logged into the node-image host while the probe pulled from the mirror host -> probe rejected, GPU needlessly disabled. It now logs into every DISTINCT host across the node image and the probe image. (The earlier "Static analysis" red was a transient PSGallery download corruption installing PSScriptAnalyzer -- infra flake, cleared by re-running CI; no code cause.) Full scripts/tests: 557 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): sanity-check pulled GPU images + full pull-path preflight (Bugbot) (#616) - Pulled node images bypassed the sanity check (Med): Confirm-GpuImagePullable accepted any successful pull, so a mis-tagged/broken mirror image enabled GPU and then aborted k3d cluster-create. It now runs Test-GpuImageRunsK3s on the pulled image (symmetric with the local build) -> CPU fallback on a broken image. - GPU preflight missed required hosts (Med): the pull-path preflight probed only $K3S_CUDA_IMAGE's host. nvcr.io (the baked device-plugin image, pulled at runtime on BOTH paths) is now probed whenever GPU is enabled, and the pull path probes every DISTINCT host across the node image AND the probe image (they can differ when both overrides are set). All SOFT (warn -> CPU fallback), never a hard fail. Full scripts/tests: 559 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): a null docker-build exit code is not a failure (Bugbot) (#616) With redirected stdout/stderr, $proc.ExitCode can remain $null after the process exits (even though Wait-ProcessWithDeadline calls WaitForExit()), so `$proc.ExitCode -ne 0` misclassified a SUCCESSFUL GPU build as failed and silently dropped GPU. Mirror the #611 k3d-create idiom: fail only on a CONFIRMED non-zero exit; a null code defers to the k3s sanity check (Test-GpuImageRunsK3s), which is the authoritative "did the build produce a working image" success marker. A null exit + working image now enables GPU; a null exit + broken image still falls back to CPU. Full scripts/tests: 561 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): GPU node-verification + fallback correctness (Bugbot) (#616) Four findings surfaced on the develop-merge review: - GPU verification always read zero (High): Confirm-GpuNode parsed the whole allocatable map (`jsonpath='{...allocatable}'` renders Go's map[...] with no JSON quotes), so the "nvidia.com/gpu" regex never matched -> every healthy GPU read as 0. Combined with the new 0-count fallback that reverted ALL GPU installs to CPU. Fixed: target the scalar field directly (jsonpath '...allocatable.nvidia\.com/gpu'). - Plugin failure left GPU requests enabled (High): when Install-GpuDevicePlugin returned false, K3D_GPU_FLAG stayed set and Confirm-GpuNode was skipped, so the chart still requested GPUs -> stranded jobs. Now a real plugin failure (flag still set) clears the flag -> CPU fallback. - K8S_VERSION=latest retained the flag (Med): the latest branch recorded a reason but didn't clear K3D_GPU_FLAG that Install-NvidiaContainerToolkit may have set -> a stock 'latest' cluster could get --gpus=all without the CUDA image. Now cleared. - Fast-path read stale GPU values (Low): Test-HealthyClusterGpuConsistent read local values.yaml, which is only clientId-healed on the adopted path. Now reads the LIVE Helm release values (helm get values) instead. Full scripts/tests: 564 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): disable NVIDIA_REQUIRE_CUDA so an older-but-valid driver enables GPU (#616) Real-box test (RTX 4050, driver 532.10 = CUDA 12.1) showed GPU falling back to CPU. Root cause: the CUDA base image bakes NVIDIA_REQUIRE_CUDA=cuda>=12.4, and the NVIDIA container runtime REFUSES to start the container when the driver is older ("unsatisfied condition: cuda>=12.4"). So a perfectly good GPU on a slightly older driver read as "Docker can't expose the GPU" and dropped to CPU. The probe and the k3s node run nvidia-smi / k3s, NOT CUDA workloads -- the real CUDA/driver compatibility for training is enforced per-pod by each training image -- so this requirement gate is meaningless for us. Disable it: - Confirm-DockerGpu probe: docker run ... -e NVIDIA_DISABLE_REQUIRE=1 - node image: ENV NVIDIA_DISABLE_REQUIRE=1 (so the k3s node boots under --gpus on an older driver); base64 embed regenerated. Now GPU enables on any driver that can actually pass the GPU through, regardless of the base image's CUDA version. Full scripts/tests: 566 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): key GPU-image reuse on build-input content hash, not just the tag (Bugbot) (#616) Adding NVIDIA_DISABLE_REQUIRE didn't change the image tag, and Build-GpuNodeImage reused any cached image that merely printed a k3s version -- so a stale pre-fix image would be reused and still fail cluster-create on an older driver. Stamp a short content hash of the build inputs (embedded Dockerfile + device-plugin manifest) as a `tracebloc.k3s-cuda-content` label at build time, and on reuse require that label to match the CURRENT hash (plus the k3s sanity check). Any change to the build inputs now busts the cache -> a stale image is rebuilt, not reused. Generalizes beyond this one fix to any future Dockerfile change. Full scripts/tests: 568 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): fast-path retries GPU + image sanity check exercises --gpus (Bugbot) (#616) - Healthy fast path prevented GPU retries (Med): a completed CPU-fallback install exited "nothing to do" before Find-Gpu/probe, so re-running after fixing Docker/driver/registry could never enable GPU -- contradicting the fallback guidance. Find-Gpu now runs before the fast path, and when an NVIDIA GPU is present but the running cluster is CPU-only (Test-RunningClusterGpuCapable) the fast path falls through to the full walk to retry GPU instead of exiting. - Image sanity check skipped GPU startup (Med): Test-GpuImageRunsK3s ran without --gpus, so a stale mirror image lacking NVIDIA_DISABLE_REQUIRE passed the check and then failed k3d cluster-create on an older driver (aborting instead of CPU fallback). It now runs WITH --gpus (no bypass), exercising the exact requirement gate cluster-create hits -> a stale image drops to CPU cleanly. Full scripts/tests: 571 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): fast-path GPU-live check uses allocatable GPU + bound nvidia-smi (Bugbot) (#616) - Fast path mistook a CUDA image for a working GPU (Med): Test-RunningClusterGpuCapable keyed on the node IMAGE name, but a CUDA node with a failed device plugin has 0 GPUs -- so a CPU-fallback re-run would wrongly take the fast path and never retry. It now checks the node's LIVE allocatable nvidia.com/gpu (the same authoritative signal Confirm-GpuNode uses), bounded with --request-timeout. - Unbounded nvidia-smi in Find-Gpu (Med): Find-Gpu now runs before the fast path, and its driver/VRAM nvidia-smi probes had no deadline, so a wedged driver could hang every re-run. Extracted Invoke-BoundedProcess (generic bounded+killable child; Invoke-DockerCli is now a thin wrapper over it) and Confirm-NvidiaDriver uses it for nvidia-smi (15s). Full scripts/tests: 572 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): fast path reconciles GPU both directions -- enable AND CPU-drift (Bugbot) (#616) The prior fix only handled "GPU present but node CPU-only -> retry". The reverse also leaked: when the node now advertises a GPU (e.g. a delayed device-plugin recovery) but the live release still requests CPU, the fast path exited "nothing to do" and training silently stayed on CPU. The fast path now shortcuts ONLY when GPU is FULLY consistent -- the node advertises a GPU AND the live release requests one (Test-RunningClusterGpuCapable + Test-LiveReleaseRequestsGpu). Any other combo on a GPU machine falls through to the full walk, which reconciles GPU_REQUESTS in either direction. Extracted the live-release GPU check into Test-LiveReleaseRequestsGpu (shared with Test-HealthyClusterGpuConsistent). Full scripts/tests: 572 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gpu): real GPU on Windows/WSL2 via CDI (validated on hardware) (#616) Validated end-to-end on a live RTX 4050 laptop (driver 532.10, Docker Desktop/WSL2): a pod that REQUESTS nvidia.com/gpu now schedules AND runs CUDA compute (vectorAdd "Test PASSED"). Previously GPU always fell back to CPU on Windows. Why the old approach couldn't work, and what replaces it: * The NVIDIA k8s DEVICE PLUGIN cannot work on Docker Desktop/WSL2: the GPU is paravirtualized (/dev/dxg + a WSL driver store), so nvmlInit() returns ERROR_NOT_SUPPORTED and the plugin registers 0 GPUs. Because it OWNS the nvidia.com/gpu extended resource it also pins the node at 0, stranding jobs. -> the node image no longer ships it (RuntimeClass only), and the installer advertises nvidia.com/gpu itself via a node-status patch (Set-NodeGpuCapacity). * CUDA itself works fine through CDI. The node image now runs nvidia-container-runtime in CDI mode and generates the WSL CDI spec AT BOOT (tracebloc-cdi-boot.sh) -- it must be at boot because the driver-store path contains a per-machine hash. * Critical gap found: `nvidia-ctk cdi generate --mode=wsl` OMITS libdxcore.so (it lives in the standard lib path, not the driver store). Without it libcuda loads but can't reach /dev/dxg, surfacing as a misleading "CUDA driver version is insufficient for CUDA runtime version". The boot script injects it. CPU and Linux are unaffected: the boot script is a strict no-op without /dev/dxg, the installer only takes the CDI path when /dev/dxg exists in the node (otherwise the standard device-plugin path is unchanged), and every step failure-falls back to CPU. Follow-up (separate repo/PR): jobs-manager must set NVIDIA_VISIBLE_DEVICES=nvidia.com/gpu=all on GPU training pods. +8 tests; full scripts/tests 580 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): node self-heals GPU capacity + verify the CDI spec (Bugbot) (#616) - GPU capacity lost after node restart (HIGH): nvidia.com/gpu was advertised by a one-shot install-time status patch, but a manually patched extended resource is NOT durable -- the kubelet re-reports node status on start, so a Docker Desktop or Windows restart dropped it while the chart still requested a GPU, stranding every job Pending until the installer was re-run. There's no device plugin to own the resource on WSL2, so the NODE now re-asserts it: the boot script (the entrypoint, so it runs on every start) backgrounds a reconciler that waits for the local API and re-patches whenever the capacity is missing/0. A reboot self-heals with no user action. Interval override: TRACEBLOC_GPU_RECONCILE_SECS. - CDI success wasn't verified (MED): the WSL2 path printed "GPU acceleration enabled" after only the capacity patch. Since the boot script guards every step with `|| true`, a failed `nvidia-ctk cdi generate` would leave us advertising a GPU pods can't use -- jobs schedule then fail CUDA with no cluster-level signal. Now the installer verifies /etc/cdi/nvidia.yaml exists and is non-empty in the node before claiming success, else CPU fallback with a clear reason. Both stay CPU-safe: the reconciler is backgrounded + fully guarded (can never delay or block k3s) and only runs when /dev/dxg exists; the spec check failure-falls back to CPU. +2 tests; full scripts/tests 582 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gpu): chart wiring for the CDI device selector + path-correct GPU guidance (#616) Completes the GPU-on-Windows chain: the installer now tells the chart to thread the CDI device selector into GPU training pods, which jobs-manager consumes as NVIDIA_VISIBLE_DEVICES (tracebloc/client-runtime#291). Without this the node advertises a GPU and pods schedule, but CUDA fails inside them. Chart: documented env.GPU_VISIBLE_DEVICES in values.yaml + values.schema.json. No template change needed -- jobs-manager-deployment's generic env pass-through already forwards it to both containers (verified with `helm template`: present when set, absent when unset/empty). Chart 1.9.27 -> 1.9.28 (version + appVersion in lockstep). Installer: $GPU_DEVICE_SELECTOR is set ONLY on the WSL2/CDI path and written as env.GPU_VISIBLE_DEVICES (also forced via --set-string on the adopted-reuse reconcile, so a stale value can't survive). Empty on a normal device-plugin (Linux) node, where the plugin owns NVIDIA_VISIBLE_DEVICES and sets concrete GPU UUIDs -- forcing a CDI selector there would break device resolution. Also fixes two misleading-guidance findings (Bugbot): - The 0-GPU skip reason always blamed the NVIDIA device plugin and told operators to mirror nvcr.io/nvidia/k8s-device-plugin. On the WSL2/CDI path there IS no device plugin, so that was a dead end. Now branches per path. - The doctor's GPU smoke-test command omitted runtimeClassName: nvidia (pods only get the GPU under it), so running it on a WORKING cluster looked like failure. It now carries the RuntimeClass, and on WSL2/CDI suggests a CUDA workload instead of nvidia-smi -- verified on real hardware, nvidia-smi fails in a pod on WSL2 (NVML unsupported) even when CUDA compute works. +4 tests; full scripts/tests 586 pass / 0 fail; check-style, check-facts, helm lint green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): remove a leftover device plugin, correct the CDI cause, collapse SERVERS (Bugbot) (#616) - CDI path left the device plugin fighting (HIGH): a device plugin OWNS the nvidia.com/gpu extended resource and on WSL2 re-reports 0 on every sync, so a DaemonSet left behind by an older install (or by a run where the /dev/dxg probe transiently missed and we took the plugin path) would overwrite the capacity patch and keep GPU disabled across ALL re-runs. The CDI path now deletes it first (--ignore-not-found, so the normal case is a clean no-op) before advertising. - Wrong CPU-fallback reason on CDI (MED): when Set-NodeGpuCapacity failed, no GPU_SKIP_REASON was set, so the caller's generic fallback filled in a device-plugin failure -- the wrong cause on a path that never uses the plugin. Set it here. - GPU mode ignored SERVERS (MED): AGENTS was collapsed to 0 but SERVERS was left alone, and EVERY server node runs the boot reconciler and advertises nvidia.com/gpu=1 for the SAME physical card -- so SERVERS>1 offered N GPUs for one device. GPU mode now forces servers=1 too, overriding an explicit value loudly. +3 tests; full scripts/tests 589 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): ship CDI setup as a k3d entrypoint DROP-IN -- the image ENTRYPOINT never ran (#616) Root cause of "GPU couldn't be wired into the cluster (CDI spec missing)" on a real Windows box. k3d does NOT use the node image's ENTRYPOINT: it replaces it with its own /bin/k3d-entrypoint.sh, which runs every /bin/k3d-entrypoint-*.sh drop-in and then execs k3s. Confirmed on the live node: Entrypoint = ["/bin/k3d-entrypoint.sh"] /bin/k3d-entrypoint-{cgroupv2,dns,mounts}.sh <- k3d's own drop-ins /usr/local/bin/tracebloc-cdi-boot.sh <- ours, never invoked /etc/cdi <- missing (spec never generated) So the CDI spec was never created and the installer correctly fell back to CPU (the verification added in c37e133 is what surfaced this instead of silently "succeeding"). Fix: install the script as /bin/k3d-entrypoint-tracebloc-cdi.sh -- k3d's documented extension point -- and restore the stock ENTRYPOINT ["/bin/k3s"]. As a drop-in it must RETURN rather than exec k3s (k3d does that), and must always exit 0: k3d runs drop-ins with `|| exit 1`, so a non-zero exit would abort the whole node, and GPU is optional. The capacity reconciler is now fully detached (</dev/null, output to /dev/null) since the drop-in exits immediately after forking it. Also corrected the Dockerfile header, which still claimed the image bakes in the device plugin (it ships only the RuntimeClass now). +2 regression tests (drop-in path + stock ENTRYPOINT; returns and never execs k3s); full scripts/tests 591 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): pass the node capacity patch via --patch-file (PS 5.1 ate the JSON quotes) (#616) Root cause of "Couldn't advertise GPU capacity on the node" on a real Windows box. Windows PowerShell 5.1 does not preserve embedded double quotes when it builds a native command line, so kubectl patch node N --type=json -p '[{"op":"add","path":...,"value":"1"}]' reached kubectl as [{op:add,...}] -> invalid JSON -> the patch failed on EVERY run. The same command works by hand only because each quote is escaped as \" there. (This is why the CDI spec now generates fine but capacity never appeared.) Fix: write the patch to a temp file and use --patch-file, which sidesteps the shell entirely. UTF8 without BOM (a BOM breaks kubectl's JSON parse). Also retried 6x/5s: the node object can still be settling right after cluster-create, and a transient 404 there would drop an otherwise-working GPU to CPU for the whole run. Temp file always cleaned up. +1 regression test (asserts --patch-file, no inline -p, BOM-less encoding, retry loop); full scripts/tests 592 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): print the doctor's GPU test with escaped quotes so it can actually be pasted (#616) Same PowerShell quote-stripping class as the capacity patch, now in user-facing guidance: the doctor printed `--overrides='{"spec":...}'`, which dies with "error: Invalid JSON Patch" when pasted into Windows PowerShell (confirmed on a live box). A suggested command that can't be run is worse than none, so the JSON is now emitted with every quote escaped as \" -- the form proven to work by hand. Verified the rendered string is paste-able: --overrides='{\"spec\":{\"runtimeClassName\":\"nvidia\", ... }}' full scripts/tests 592 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): libdxcore injection missed the generator's indentation (real CUDA failure) (#616) Root cause of "CUDA driver version is insufficient" inside GPU pods on a real box. Confirmed live: the CDI spec existed and parsed (`nvidia-ctk cdi list` -> 1 device), but `grep libdxcore /etc/cdi/nvidia.yaml` found NOTHING -- the injection never happened. The awk anchor was `^ mounts:$` (exactly two spaces), while nvidia-ctk marshals YAML with yaml.v3 at FOUR spaces, so it never matched. Without libdxcore, libcuda loads but can't reach /dev/dxg -> that misleading driver error, on a node that advertises a healthy GPU. Fixes: - Indent-agnostic anchor (`^[[:space:]]*mounts:[[:space:]]*$`) and the inserted item now MIRRORS the generator's own first-item indentation instead of hardcoding it. Hardcoding was doubly wrong: YAML forbids mixing indents in one list, so a fixed 4-space item next to 2-space items would have made the whole spec unparseable (CDI then injects nothing -- same symptom, different cause). Verified both 2- and 4-space specs parse with libdxcore present, and that the old awk produced invalid YAML on a 2-space spec. - The edit is only adopted if `nvidia-ctk cdi list` still parses the result; otherwise the original spec is restored (GPU without libdxcore beats a silently disabled GPU). - The installer now verifies libdxcore is IN the spec (not merely that a spec exists) and falls back to CPU with a specific reason if not -- so this class of silent miss surfaces at install time instead of as a confusing CUDA error later. +2 regression tests; full scripts/tests 594 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): never advertise a GPU that CDI can't back, + honest GPU messaging (Bugbot) (#616) - Reconciler advertised GPU without CDI (HIGH): the boot reconciler re-asserted nvidia.com/gpu whenever /dev/dxg existed, even if `cdi generate` failed or left an incomplete spec. The installer already refuses in that case, but a Docker Desktop or Windows restart re-ran the reconciler and put capacity back onto a node where injection is broken -- pods schedule, then fail CUDA with no cluster-level signal. The reconciler now applies the SAME standard (cdi_ok: spec non-empty AND contains libdxcore AND `nvidia-ctk cdi list` parses) and simply doesn't start otherwise. - libdxcore path was fragile (MED): hardcoded /usr/lib/x86_64-linux-gnu/libdxcore.so, but it lives elsewhere on other Docker Desktop / WSL2 versions (/usr/lib/wsl/lib, the WSL driver store). A miss silently skipped the injection while the spec still passed the non-empty check -> GPU advertised, CUDA fails with the misleading driver error. Now probes the known locations, falls back to the linker cache, and mounts it at the path where it was actually found. - Green success before verification (MED): `Ok "GPU acceleration enabled (WSL2/CDI)"` printed at the capacity patch, before Confirm-GpuNode confirms allocatable GPU -- so a failed verification showed a green enabled line followed by a CPU fallback. Now an Info "verifying..." line; Confirm-GpuNode's "GPU verified and available" is the only success claim. - Misleading preflight warning + stale comments (LOW): the soft GPU preflight always said the node image "can't be built", including on the pull/mirror path where nothing is built locally -- now path-aware. Also corrected three comments that still claimed the device plugin is baked into the node image (this PR ships only the RuntimeClass). +4 tests; full scripts/tests 598 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): the in-WSL toolkit step no longer claims GPU is ready (Bugbot) (#616) Install-NvidiaContainerToolkit printed a green "GPU acceleration ready", set K3D_GPU_FLAG, and CLEARED GPU_SKIP_REASON after merely verifying the toolkit inside the WSL distro. But Confirm-DockerGpu is the authoritative gate and runs later, so: - operators could see a green ready line and then a CPU fallback, and - clearing the skip reason dropped the real cause when the later gate failed. It now reports only what it established ("NVIDIA Container Toolkit present in <distro>", Info not Ok) and logs that GPU is still gated on the Docker GPU probe. It no longer touches K3D_GPU_FLAG or GPU_SKIP_REASON. Nice invariant this establishes: K3D_GPU_FLAG is now assigned "--gpus=all" in exactly ONE place -- the authoritative gate -- which a new test pins, along with the toolkit step no longer setting the flag or clearing the reason. +1 test; full scripts/tests 599 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(gpu): machine-check the smoke-test JSON instead of eyeballing it (Bugbot) (#616) Bugbot reported an "extra closing brace" in the WSL2/CDI --overrides payload. Verified FALSE POSITIVE two ways: a JSON parser accepts it with the correct structure, and this is the exact escaped form that ran on a live box (kubectl created the pod; it failed later at CUDA, not at parsing). The `}}}` after "1" is correct -- it closes limits, then resources, then the container object. No behaviour change, but the concern deserves to be checkable rather than argued: extracted the command into a PURE Get-GpuSmokeTestCommand (selector in, string out), so tests can render both variants, strip the paste-escaping and ConvertFrom-Json them. A genuine brace slip now fails CI instead of shipping a command that errors on a healthy cluster. Also removes the duplication between the two branches and gives the diagnostics bundle one source of truth. +2 tests (JSON validity for the WSL2/CDI and device-plugin variants), and the previous source-scraping assertions now go through the builder. Full scripts/tests 601 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): close version-drift gaps that would cost GPU on a different Windows machine (#616) Audit prompted by "what breaks on someone else's laptop". The reassuring part first: a version mismatch CANNOT break the install. The k3d drop-in runs under `|| exit 1`, so a failure there would abort the node and fail cluster-create -- I verified under real `dash` (the node image's /bin/sh, no bashisms) that it exits 0 with the toolkit absent, with every nvidia-ctk subcommand failing, and with a garbage spec. It degrades to CPU instead. Two gaps found that would silently LOSE GPU on another machine: 1. The NVIDIA Container Toolkit was UNPINNED (`apt-get install nvidia-container-toolkit`), so two machines built weeks apart could get different builds. The whole WSL2 path leans on version-sensitive surfaces -- `cdi generate --mode=wsl`, `config --set nvidia-container-runtime.mode=cdi`, and the exact YAML the generator emits (our libdxcore injection parses it) -- so a future release changing the spec shape would break GPU on new installs while existing ones kept working. Pinned to the build validated on hardware (1.19.1-1) via ARG NCT_VERSION, and it FALLS BACK to latest if that version has aged out of the apt repo, so a stale pin degrades to "unpinned" rather than failing the build (which would cost GPU entirely). Also logs `nvidia-ctk --version` so the image records what it got. 2. The cdi_ok gate required `nvidia-ctk cdi list` to EXIST. That subcommand is version-dependent, so on a toolkit build without it we would have refused to advertise a perfectly working GPU -- a false negative on someone else's machine. The gate now uses format-stable structural facts (spec non-empty, declares nvidia.com/gpu, exposes /dev/dxg, carries the libdxcore mount) and uses `cdi list` only as an EXTRA veto when available. Verified both directions with a dash harness: a good spec + a toolkit lacking `cdi list` now advertises the GPU; a spec missing libdxcore still does not. +2 tests; full scripts/tests 603 pass / 0 fail; dash -n, check-style, check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(gpu): every GPU-enable failure now names an actionable cause (#616) The residual risks that differ machine to machine all degraded to CPU correctly, but most collapsed into "the GPU node image build failed (docker build exit 1)" -- true, and useless to the person holding the laptop. Now each one names what happened and the single thing to do about it: - older Docker Desktop / no BuildKit labs frontend -> "update Docker Desktop, or point TRACEBLOC_K3S_CUDA_IMAGE at a prebuilt image" - full disk -> "free up space and re-run" - retired CUDA base tag -> names the tag + TRACEBLOC_CUDA_BASE_TAG - TLS-inspecting proxy (x509) -> TRACEBLOC_CA_BUNDLE - blocked/offline registry -> TRACEBLOC_IMAGE_REGISTRY (the air-gap path) - registry rate limit -> retry or use the mirror - anything else -> exit code + a pointer to the build output in the log Classification lives in a PURE Get-GpuBuildFailureReason so every branch is unit-tested. Also: - The Docker GPU probe failure now quotes the DETECTED driver version and a concrete minimum ("update to 525 or newer"). Our install gate accepts 460+, but WSL2 CUDA needs much newer, so "update your driver" left people unable to tell if theirs qualified. - GPU-detected-but-not-enabled gets its OWN summary block instead of being crammed into the Mode line: what happened, why, "fix that then re-run", and the log path. Previously the reason was a long parenthetical that was easy to miss. +12 tests; full scripts/tests 615 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): no green line may claim GPU is enabled before verification (Bugbot) (#616) Third report of this defect, so I swept for the whole family instead of patching one line. Fixed the reported one: the top-level gate printed Ok "GPU enabled -- cluster will use the custom k3s-CUDA image with --gpus=all" at Step 2, before cluster-create, the node's CDI wiring, and Confirm-GpuNode -- any of which can still clear K3D_GPU_FLAG. Now Info, worded as intent ("GPU support prepared … verified once the node is up"). The sweep found TWO MORE instances on the device-plugin path (pre-existing, not introduced here, but the same defect and the same misleading sequence): Ok "GPU acceleration enabled." both when the DaemonSet already existed and after a successful rollout -- printed before Confirm-GpuNode, which can find 0 allocatable GPUs and fall back to CPU. Both are now Info lines that say verification is pending. Message-only; no behaviour change on that path. Established the invariant instead of relying on review: Ok is reserved for facts already established, Info for intent. Exactly two Ok-level GPU lines remain, and both state something verified -- the image was built AND passed its k3s sanity check, and the node WAS observed advertising a GPU. A test pins the count at 2, asserts neither says "enabled", and asserts the four pre-verification lines are Info. +3 tests; full scripts/tests 618 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): the CDI revert must not require `cdi list` either (Bugbot) (#616) My own inconsistency, caught correctly. I made the cdi_ok gate tolerate a missing `nvidia-ctk cdi list` (it's version-dependent), but left the REVERT one block earlier calling it unconditionally. On a toolkit without that subcommand the sequence was: libdxcore injected -> `cdi list` "fails" (absent) -> edit reverted -> spec now lacks libdxcore -> installer reports "the spec is missing libdxcore" -> CPU i.e. a perfectly good injection thrown away, and a reason that is factually wrong (the library WAS found and mounted) and unactionable. The revert now reverts only when the parser is AVAILABLE and actively REJECTS the result -- the same rule as cdi_ok. Verified with a dash harness: with `cdi list` absent, libdxcore survives in the spec, the spec is still valid YAML, and the reconciler advertises the GPU. +1 test asserting every `cdi list` use is availability-gated: two probes, exactly two executable bare uses (comments excluded), and both guarded shapes pinned. Full scripts/tests 619 pass / 0 fail; dash -n, check-style, check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(gpu): shell-runnable drift guard for the embedded node-image files (review #633) Review asked for a check-facts/bats-style guard on the base64 embeds, having verified they are byte-identical today but noting nothing enforces it: check-facts guards the version PINS, not whole-file embedding, and Get-GpuBuildContentHash only hashes the EMBEDDED copies for local-build caching -- it never compares them to the repo files. A future edit to docker/k3s-cuda/* that isn't re-embedded would silently diverge (published image vs the installer's local build) with nothing failing. A Pester suite already asserts exactly this and runs in CI on windows-latest AND ubuntu-latest, so drift was in fact caught -- the reviewer simply had no pwsh to see it. That is a fair ergonomics gap, so this adds the bash-side mirror: decode each of the three blobs and diff against docker/k3s-cuda/*, plus non-empty checks (a truncated re-embed) and the drop-in shape invariant (no `exec /bin/k3s`, always `exit 0` -- the mistake that shipped once). CI already runs `bats scripts/tests/*.bats`, so it is picked up automatically. Verified the guard actually bites: appending a line to nvidia-runtimeclass.yaml without re-embedding turns test 2 red, and restoring it returns 5/5 green. On failure it prints the diff and the exact remedy (re-embed + regenerate the manifest). bats-hygiene passes (every assertion carries `|| return 1`, per #527). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): quote whitespace args + fail fast when GPU hosts are known-blocked (Bugbot) (#616) Two of four findings were real; the other two are refuted with evidence in the threads. 1. (Med) Arguments with whitespace could split. Invoke-BoundedProcess joins args into ONE command line, so an unquoted value containing a space -- a registry username, or a temp path under a profile like "C:\Users\First Last\..." -- silently became two arguments and corrupted the command. Now quotes any arg containing whitespace, leaves already-quoted values alone (so call sites that quote themselves aren't double-quoted), and preserves an empty string as a present-but-empty argument. Fixes the class, not just docker login. 2. (Med) A re-run on a restricted network looked hung. Preflight already soft-probes the GPU download hosts, but the gate then ran the probe (180s) and the build/pull (up to 15-20 min) anyway, so the "re-run to retry GPU" advice we print led to minutes of timeouts before the CPU fallback. Preflight now records the unreachable host and the gate short-circuits on it with an actionable reason (set TRACEBLOC_IMAGE_REGISTRY / a prebuilt image). Defaults empty, so a reachable machine is unaffected. +5 tests (incl. behavioural joiner cases: space-containing value, already-quoted path, empty string). Full scripts/tests 625 pass / 0 fail; embed-drift bats 5/5; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): only path-required hosts may skip GPU -- don't break air-gapped mirrors (Bugbot) (#616) A regression I introduced one commit earlier. The fast-fail short-circuit treated EVERY soft GPU host probe as blocking, but on a mirror / prebuilt-image install nvcr.io is unreachable BY DESIGN -- the node image and the CDI probe image are both re-homed to the mirror. So the "don't waste minutes timing out" optimisation disabled GPU for exactly the air-gapped case the mirror exists to serve, and the remedy told the operator to configure the mirror they had already configured. Probes now carry gpuBlocking, set only for the hosts the CURRENT path actually needs: - build path: nvcr.io + nvidia.github.io -> blocking - mirror/prebuilt path: the configured GPU registry host -> blocking; nvcr.io still probed (a warning is useful) but NON-blocking Only a blocking probe arms GPU_HOSTS_UNREACHABLE. The skip remedy is now path-aware too: on the mirror path it says to check that the configured GPU registry is reachable and holds the k3s-CUDA image, rather than suggesting a mirror. +2 tests pinning both directions. Full scripts/tests 627 pass / 0 fail; check-style + check-facts green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
release-train: develop -> staging
LukasWodka
commented
Aug 10, 2026
ContributorAuthor
bugbot run |
Uh oh!
There was an error while loading. Please reload this page.
* ci: enroll version-bump-gate (backend#1563) * ci: re-run version-bump-gate on label events (skip-version-gate override; Bugbot) * ci: restore caller + add labeled/unlabeled triggers (fix empty-file slip; Bugbot skip-label)
LukasWodka
commented
Aug 10, 2026
ContributorAuthor
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
2 issues from previous reviews remain unresolved.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 6758867. Configure here.
release-train: develop -> staging
LukasWodka
commented
Aug 10, 2026
ContributorAuthor
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
2 issues from previous reviews remain unresolved.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit c73b256. Configure here.
LukasWodka
commented
Aug 10, 2026
ContributorAuthor
Ticket+ship (Medium): tracked in #656. |
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated promotion by the release train (RFC-0008 D14). Head is the train-managed
release-train/to-mainbranch (a mirror ofstaging), so it never collides with a human PR. Merged only when the fr-gate is green.Note
High Risk
Production chart promotion touching CronJob upgrade/refresh logic, MySQL startup guards, hostPath permission init, and new GPU node image paths—any regression affects live edges on upgrade or restart.
Overview
Automated staging → main promotion for the client Helm chart (1.9.15 → 1.9.28), bundling edge-installer and chart behavior that already landed on
develop/staging.GPU edges (#616): Adds
docker/k3s-cuda(CUDA Ubuntu + pinned k3s, NVIDIA Container Toolkit in CDI mode,nvidiaRuntimeClass, WSL boot script for CDI +nvidia.com/gpureconciliation) plus a manualbuild-k3s-cudaworkflow andcheck-factsdrift rows soK3S_TAG/CUDA_TAGstay aligned withfacts.envand the Windows installer.Chart runtime changes:Auto-upgrade jobs get a computed
activeDeadlineSecondsfloor so hunghelm repowork cannot block future ticks. Image-refresh builds an explicit in-cluster kubeconfig, adds flap detection (maxRefreshAttempts), bounded API calls, and safer annotation handling. jobs-manager on hostPath runs a privileged init to make shared/logs writable; memory requests are raised to match limits for OOM resilience (#1144). MySQL gains a format-guard init against 5.7 ↔ 8.x datadir mismatches. GPU env vars render only whenGPU_LIMITSis set;GPU_VISIBLE_DEVICESis documented for CDI/WSL. requests-proxy gets TCP probes. Opt-inserviceDbAccountsaddstb_meta/tb_ingestsecrets and env (default off).Repo hygiene: New
Makefile(check/check-all/setup), synced org standards inCLAUDE.md, action-pins enforced in code-quality, version-bump-gate ondevelopPRs, public PII gate workflow removed; installer-tests also runs whendocker/k3s-cuda/**changes.Reviewed by Cursor Bugbot for commit c73b256. Bugbot is set up for automated code reviews on this repo. Configure here.