fix(installer): stop k3s version pin from silently drifting (#547) by shujaatTracebloc · Pull Request #565 · tracebloc/client

shujaatTracebloc · 2026-08-03T13:32:56Z

What & why

Root-caused from the Windows stuck-install incident (#545/#547): the affected client's node ran k3s v1.35.5+k3s1 while the installer pin is v1.29.4-k3s1 — a version the chart was never validated against, which widened the timing window the RWO deadlock (#549) exploited.

The post-incident audit found the pin can drift through four compounding gaps. This PR closes three; F3 is de-risked and tracked in #547.

Changes

F1 — misleading docs + silent latest float. The header docs advertised default: latest, inviting users to set K8S_VERSION=latest, which skips --image and floats to k3d's bundled default k3s. Fixed the docs in both installers and added a loud warning at create time when latest is used (bash + PowerShell).
F2 — reuse path never re-checked the version. A cluster born unpinned (older installer, latest, or a manual k3d create) kept its k3s forever across later correctly-pinned re-runs — the single best explanation for the observation. Added _check_existing_cluster_k8s_version (bash) and a parity check in New-K3dCluster (PowerShell): warn + recreate remedy on drift, mirroring the existing proxy/CA/storage drift checks. This is what surfaces a drifted cluster (like the incident client's) on its next installer run.
F4 — CI blind spot.check-facts.sh compared only the pinned version strings, not that the create command actually wires the pin. Added a structural guard asserting --image rancher/k3s: is present in cluster.sh and install-k8s.ps1, so a refactor can't silently unpin k3s while CI stays green.

F3 (Windows winget installs an unpinned k3d) is intentionally not in this PR: with --image now guaranteed on create, k3d's own version can no longer float k3s, so F3 is de-risked to a k3d-CLI-determinism nicety — tracked as a checklist item in #547.

Tests (added only)

scripts/tests/cluster.bats: +7 for _check_existing_cluster_k8s_version (empty/latest no-op, match, drift → warn, registry+digest tag parse, unparseable ref, docker-fails).
scripts/tests/install-k8s.Tests.ps1: +5 source guards (create --image, latest warning, reuse inspect+compare, recreate remedy, docs no longer say default: latest).
scripts/tests/check-facts.bats: fixture extended additively with the wiring line so the new F4 guard is exercised.

Local: cluster.bats 74/74, check-facts.bats 13/13, Pester 387/0/9, shellcheck --severity=error clean, bash -n clean, PS parses clean, PSScriptAnalyzer adds nothing.

Closes#547

Note

Medium Risk
Touches core cluster creation and reuse paths in bash/PowerShell installers; behavior is warn-only on drift but changes defaults and CI gates for k3s pinning.

Overview
Hardens the local installer so the validated k3s pin is actually used at cluster create time and drift on reused clusters is visible instead of silently reused (#547).

Create path: Bash and PowerShell now document the pinned default (not latest), warn loudly when K8S_VERSION=latest, and pass --image rancher/k3s:<pin> on fresh k3d creates. Reuse / healthy fast-path: New checks compare the running node image to the pin and print a non-fatal recreate hint (including when install would otherwise exit on “already healthy”). check-facts.sh adds a wiring guard so CI fails if the create-time --image literal is removed from cluster.sh / install-k8s.ps1, with messaging that --write cannot fix wiring gaps.

Docs:INSTALL.md adds idempotent hostpath staging steps for laptop installs (PowerShell robocopy / Unix cp -R). Tests cover drift checks, assess healthy-path behavior, check-facts wiring failures, and PowerShell source guards; installer script hashes in manifest.sha256 are updated.

^{Reviewed by Cursor Bugbot for commit 8ba7cb6. Bugbot is set up for automated code reviews on this repo. Configure here.}

Root-caused from the Windows stuck-install incident: a client ran k3s v1.35.5 while the pin was v1.29.4-k3s1. Addresses three of the four compounding gaps the audit found (F3 left as a tracked checklist item): - F1: the header docs advertised `default: latest`, inviting users to set K8S_VERSION=latest, which floats to k3d's bundled default k3s. Fix the docs in both installers, and warn loudly at create time when `latest` is used. - F2: the reuse/adopt path never re-checked the running node's k3s version, so a cluster born unpinned (old installer / latest / manual create) persisted forever across later correctly-pinned re-runs — the single best explanation for the observation. Add _check_existing_cluster_k8s_version (bash) and a parity check in New-K3dCluster (PowerShell): warn + recreate remedy on drift. - F4: check-facts.sh only compared the pinned version STRINGS, not the create wiring, so `--image rancher/k3s:` could be dropped while CI stayed green. Add a structural guard asserting the pin is wired in cluster.sh + install-k8s.ps1. With --image now guaranteed on create, k3d's own version no longer floats k3s, so F3 (winget installs unpinned k3d) is de-risked and tracked in #547. Tests (only added): +7 bats for _check_existing_cluster_k8s_version, +5 Pester source guards, check-facts.bats fixture extended with the wiring line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

) install-k8s.sh / cluster.sh / install-k8s.ps1 hashes changed; the supply-chain R8 gate (gen-manifest.sh --check) requires the committed manifest to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…bot #565) Bugbot: the drift check only lived on the full reuse path (_handle_existing_cluster / New-K3dCluster), but both installers short-circuit earlier when a re-run classifies as healthy (bash assess_existing_install, PS completed+healthy fast-path). A healthy-but-drifted cluster — the #547 STEADY STATE — would hit "already set up / nothing to do" and never see the warning, exactly the population the check is meant to help. - bash: assess_existing_install's healthy branch now calls _check_existing_cluster_k8s_version before the handoff (guarded by declare -F). - PS: extracted the inline reuse-path check into Test-K3sVersionDrift and call it from BOTH New-K3dCluster and the completed+healthy fast-path in main. Tests: +2 assess.bats (healthy runs it; --force skips it); Pester #547 block updated to assert the shared function + both call sites. Manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…run --write" (Bugbot #565) Bugbot: the F4 wiring guard incremented the same `drift` counter as version-string mismatches, so a missing create-time --image pin ended with "fact(s) drifted... Run 'check-facts.sh --write'". But --write only restamps version strings and cannot restore create-time wiring — the summary pointed developers at a no-op fix. Track wiring failures in a separate counter and emit a wiring-specific message (this is a WIRING gap; restore the --image rancher/k3s:${K8S_VERSION} flag by hand). +1 check-facts.bats: a missing --image pin fails with the WIRING message and never the --write hint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…Linux (#547) A client re-running the dataset-copy step hit "already exists" from non-idempotent `mkdir` + `Copy-Item -Recurse`. The repo only documented the Linux `kubectl cp` staging path, with no hostpath/Windows guidance. Add an idempotent hostpath staging section: Windows uses `New-Item -Force` + `robocopy /E` (merges into an existing target, safe to re-run); macOS/Linux use `mkdir -p` + `cp -R`. Notes the plain-mkdir "already exists" error is harmless (data already staged). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Bugbot: _check_existing_cluster_k8s_version (bash) and Test-K3sVersionDrift (PS) ran a bare `docker inspect` with no deadline, and both healthy fast-paths now call them — a wedged Docker engine could hang a headless "already healthy" re-run AFTER success was printed, violating the installer's bounded-probe rule. - bash: wrap the inspect in _bounded (timeout/gtimeout; 124 on timeout → the existing `|| return 0` makes it a silent no-op). - PS: run it via Start-Job + Wait-JobWithProgress -TimeoutSec 15 (mirrors Test-ClusterRunning); on timeout, skip the check with a log line. Tests: cluster.bats setup overrides _bounded so the docker shell-function mock is exercised on Linux CI too (timeout can't exec a function); Pester asserts the bounded Start-Job pattern tied to the "Checking k3s version" probe. Manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9c501f0. Configure here.}

…hell (Bugbot #565) Bugbot: the remediation hint told devs to restore `rancher/k3s:${K8S_VERSION}` in both files, but the PowerShell guard matches the fixed string `rancher/k3s:$K8S_VERSION` (no braces) — following the hint in the PS create path would leave CI red even though --image is correctly wired. Reword the hint to name BOTH shell forms (bash cluster.sh uses ${K8S_VERSION}; PowerShell install-k8s.ps1 uses $K8S_VERSION) and point at the exact literal each ✖ line already prints. +2 assertions in check-facts.bats locking both forms. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

LukasWodka · 2026-08-03T17:06:29Z

/fr-pass

…drift (backend#2253) Two Bugbot findings on the CLI-only upgrade path: - Failed update no longer reports success. install_tracebloc_cli is non-fatal (written for a client that is already connected), so a failed download would exit 0 and leave the update nag in place while `tracebloc upgrade` looked like it worked. On THIS path the CLI update is the whole job, so verify it: when TB_CLI_LATEST is known and the CLI is still behind it afterward, warn and exit non-zero (telemetry then records failed, not succeeded). - Surface the same k3s-drift (#547/#565) and GPU-consistency (client#835) advisories the healthy hand-off prints — this path also exits before _handle_existing_cluster, so a drifted-but-healthy cluster would otherwise get no signal on upgrade. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ackend#2253) (#864) * fix(installer): tracebloc upgrade updates a healthy-but-behind CLI (backend#2253) The stop-and-check gate hands a verifiably-healthy machine straight to the home screen ("already set up — no need to run the installer again") and updates nothing. Its CLI floor stops at 0.10.0 (below = mandatory reinstall), but the CLI's own update nudge fires against the latest release — so a CLI at e.g. 0.10.5 with latest 0.10.8 was nagged forever while `tracebloc upgrade` (which re-runs this installer) found the box healthy and changed nothing. Bridge the two definitions without weakening the floor: - assess.sh: a new read-only _assess_cli_behind_latest, gated on TB_UPGRADE_CLI, compares the installed CLI against TB_CLI_LATEST (resolved and passed by the CLI — no network here). classify emits a DISTINCT cli-behind-latest reason, ordered AFTER the floor check so below-floor stays cli-outdated (still a mandatory full reinstall). Inert on every ordinary installer run. - install-k8s.sh main(): on cli-behind-latest, update ONLY the CLI (a small, isolated download via upgrade_cli_only) and exit — no full reconcile. assess stays a read-only classifier; the CLI-install mutation lives in main(). - install.sh: TB_UPGRADE_CLI=1 skips the bootstrap's healthy bailout so the run reaches the gate, WITHOUT forcing a reinstall. The CLI half (setting TB_UPGRADE_CLI / TB_CLI_LATEST) is in tracebloc/cli. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): upgrade_cli_only fails honestly and surfaces cluster drift (backend#2253) Two Bugbot findings on the CLI-only upgrade path: - Failed update no longer reports success. install_tracebloc_cli is non-fatal (written for a client that is already connected), so a failed download would exit 0 and leave the update nag in place while `tracebloc upgrade` looked like it worked. On THIS path the CLI update is the whole job, so verify it: when TB_CLI_LATEST is known and the CLI is still behind it afterward, warn and exit non-zero (telemetry then records failed, not succeeded). - Surface the same k3s-drift (#547/#565) and GPU-consistency (client#835) advisories the healthy hand-off prints — this path also exits before _handle_existing_cluster, so a drifted-but-healthy cluster would otherwise get no signal on upgrade. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…ories (backend#2674) (#870) * test(assess): guard every early-exit path reaches the drift/GPU advisories (backend#2674) assess short-circuits an already-set-up machine before the normal flow reaches _handle_existing_cluster, where the k3s-drift (#547/#565) and GPU-consistency (client#835/#852) advisories run. Every early-exit terminal must run both itself — a rule we kept re-learning one instance at a time (the k3s check, the GPU check, then the cli-behind-latest → upgrade_cli_only path in backend#2253, each patched only after the omission was spotted). New suite scripts/tests/assess-early-exit-drift.bats catches the CLASS: - behavioral: drives the healthy hand-off and upgrade_cli_only, asserts BOTH advisories run (and, for the hand-off, before it); - static enumeration that FAILS CLOSED on a new uncovered terminal: pins the exit-bearing functions in assess.sh (_assess_handoff) and install-cli.sh (upgrade_cli_only), pins _assess_handoff to one call site, and asserts each early-exit decision calls both advisories at the source level; - a fixture proving the enumeration actually detects an unguarded early-exit. Mutation-verified: dropping either advisory fails the behavioral + static tests; adding a new exit-bearing function fails the pin. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(assess): derive the advisory set instead of hardcoding it (backend#2674) Bugbot: the ADVISORIES pair was restated, so a new advisory added to the healthy hand-off but not upgrade_cli_only would pass both the static loop and the behavioral stubs — the same class this suite stops, on the advisory axis. Derive the set from the `declare -F X && X` guard idiom in the reference path (the healthy hand-off) and assert upgrade_cli_only runs the SAME set. Divergence in either direction now fails. Mutation-verified: a 3rd advisory on one path only fails the parity test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(assess): harden the scanner against compound exits + both guard idioms (backend#2674) Two Bugbot findings on the scanner itself — the blind spots that matter most for a class-catcher: - Exit scan matched only a leading `exit`, so a terminal written `foo && exit`, `foo; exit`, or `then exit` slipped the pin. Now matches `exit` as a word in any position — but strips single/double-quoted spans first so an embedded `awk '... exit }'` (as in _assess_cluster_servers_running) is not a false positive, and skips/strips comments. - Advisory derivation saw only the one-liner `declare -F X && X`, so a _check_ advisory added via the `if declare -F X; then X; fi` block (the form install_tracebloc_cli already uses) slipped parity. Now keys on `declare -F _check_*` in either idiom, skipping comments so a commented-out guard cannot pad the set. Removed the now-unused _funcs_calling helper. Fixture extended to every exit spelling + an embedded-awk-exit + a commented-out guard. Mutation-verified: a compound-exit terminal fails the pin; a 3rd advisory via if-then on one path only fails parity. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(assess): count hand-off invocations in any spelling, not just line-leading (backend#2674) Bugbot: the _assess_handoff call-site pin used a line-leading grep, so a second hand-off in the file OWN case-arm style (state) … _assess_handoff ;;), or via &&/then, never incremented the count — the exact inline early-exit this suite exists to catch. Add _count_calls, which counts invocations of a symbol as a word in any position (excluding the definition token and comments/quoted spans, same handling as the exit scan), and use it for the pin. Fixture now plants a case-arm one-liner hand-off and asserts the count. Mutation-verified: a syntactically-valid inline second hand-off fails the pin. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

shujaatTracebloc self-assigned this Aug 3, 2026

chore(installer): regenerate manifest.sha256 for the k3s-pin edits (#547
9bd2e41
) install-k8s.sh / cluster.sh / install-k8s.ps1 hashes changed; the supply-chain R8 gate (gen-manifest.sh --check) requires the committed manifest to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

shujaatTracebloc marked this pull request as ready for review August 3, 2026 13:38

shujaatTracebloc requested a review from saadqbal as a code owner August 3, 2026 13:38