Skip to content

fix(installer): pin the k3d release (K3D_VERSION) — deterministic, rate-limit-proof installs - #382

Merged
LukasWodka merged 8 commits into
developfrom
fix/installer-pin-k3d
Jul 24, 2026
Merged

fix(installer): pin the k3d release (K3D_VERSION) — deterministic, rate-limit-proof installs#382
LukasWodka merged 8 commits into
developfrom
fix/installer-pin-k3d

Conversation

@LukasWodka

@LukasWodkaLukasWodka commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

The upstream k3d install script resolves its version via a GitHub releases/latest redirect that breaks under GitHub rate limiting on shared egress IPs (CI runners, corporate NAT). That took down 2/9 distro CI jobs on main on 2026-07-21 (Prereqs — amazonlinux:2023, Prereqs — ubuntu:24.04) with a bare curl: (22) 404 / Failed to install k3d — and the same failure mode can hit real customer installs behind shared NAT (hospitals, enterprises).

This pins k3d the same way K8S_VERSION already pins k3s: a K3D_VERSION default (v5.9.0) in common.sh. Since review, upstream's install.sh is no longer used at all@saadqbal pulled the pinned script and found it performs no checksum verification (its downloadFile fetches the bare binary; installFile just chmod+cp), contrary to what this PR originally claimed. The installer now fetches k3d-linux-<arch>plus the release's checksums.txt directly from the pinned release and sha256-verifies before installing (fail-closed; mirrors the existing _fetch_kubectl pattern). This also removes the releases/latest lookup from the default path entirely, so the rate-limit failure can't occur there. K3D_VERSION=latest resolves the newest tag at install time via the plain /releases/latest redirect (no API) and then takes the same verified path; an empty value falls back to the pin, and malformed values fail closed before any fetch (a /-carrying value could otherwise traverse outside k3d-io/k3d — Bugbot). Both the Tier-0 (user-space) and sudo paths install via a plain verified mv.

Windows rider (same review finding):install-k8s.ps1 fetched sha256sum.txt, an asset name that has never existed on k3d releases (it's checksums.txt) — the request 404'd into a fail-open catch, so the Windows k3d verification silently never ran. URL fixed; verification is now live there too.

Also: the preflight tool-host probe for a missing k3d now points at github.com (the raw.githubusercontent.com install script is no longer fetched).

Related

Failing run: https://github.com/tracebloc/client/actions/runs/29839314271

Type of change

  • Bug fix

Test plan

  • bats scripts/tests/setup-linux.bats50/50, incl. 5 tests for the new path: verified direct download (asserts binary + checksums.txt fetched at the tag, sha256sum --check ran, no install.sh anywhere), checksum-mismatch fails closed with nothing installed, asset-missing-from-checksums fails closed, sudo install target, latest resolves then takes the same verified path
  • bats scripts/tests/preflight.bats → 52/52
  • shellcheck --severity=error clean; bash -n clean; scripts/check-style.sh clean; scripts/gen-manifest.sh regenerated
  • CI Prereqs matrix (9 distros) exercises the real download + verification end-to-end on this PR

Deployment notes

Pin bump procedure: update K3D_VERSION in scripts/lib/common.sh (+ help text/golden/manifest) when we want a newer k3d — same convention as the K8S_VERSION pin.

Checklist

  • Tests added / updated and passing locally
  • Docs updated if behavior or config changed (--help env list + script headers)
  • No secrets / credentials in the diff
  • For security-sensitive paths: appropriate reviewer requested (@saadqbal reviewing)
  • Terminal output follows STYLE.MDbash scripts/check-style.sh passes

🤖 Generated with Claude Code


Note

Medium Risk
Touches privileged installer download paths and supply-chain verification on Linux and Windows; behavior change is intentional hardening but misconfiguration or GitHub egress issues will now hard-fail k3d install instead of silently proceeding.

Overview
Linux installs no longer pipe through upstream k3d install.sh (no checksum verification, releases/latest failures under GitHub rate limits). The installer adds K3D_VERSION (default v5.9.0) and downloads k3d-linux-<arch> plus checksums.txt from that tag, SHA256-verifying before install—same pattern as kubectl. K3D_VERSION=latest resolves the tag via /releases/latest, then uses the same verified path; invalid tags abort before any fetch.

Windows direct-download path is fixed: checksums came from a non-existent sha256sum.txt URL and errors were swallowed, so verification never ran. It now uses checksums.txt and fails closed (remove binary on fetch/mismatch/missing line).

Docs/help/golden, preflight egress hint (github.com for k3d), manifest hashes, and bats coverage for pin/latest/mismatch/sudo/malformed version are updated.

Reviewed by Cursor Bugbot for commit ab9eabc. Bugbot is set up for automated code reviews on this repo. Configure here.

…latest lookup
The upstream k3d install script resolves its version via a GitHub
releases/latest redirect, which breaks under GitHub rate limiting on
shared egress IPs (CI runners, corporate NAT) and failed 2/9 distro CI
jobs on main on 2026-07-21 with a bare 'curl: (22) 404 / Failed to
install k3d'. Retrying can't help — the limit persists per-IP.
Pin it the way K8S_VERSION already pins k3s:
- K3D_VERSION (common.sh, default v5.9.0) — the install script is now
fetched AT the pinned tag (immutable bytes instead of k3d's mutable
main) and TAG=<pin> makes it skip the releases/latest lookup entirely.
Its checksums.txt verification of the binary still runs.
- K3D_VERSION="" or "latest" restores resolve-at-install-time.
- Covers both the Tier-0 (user-space) and sudo install paths.
- Help text + golden catalog + signed manifest regenerated.
Verified: scripts/tests/distro-prereqs.sh passes in fresh
amazonlinux:2023 and ubuntu:24.04 containers (the two failing CI jobs),
installing k3d v5.9.0; setup-linux.bats 47/47 incl. two new pin tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@LukasWodkaLukasWodka self-assigned this Jul 24, 2026
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

bugbot run

Comment threadscripts/lib/common.sh
Comment threadscripts/lib/setup-linux.sh Outdated
- Fail closed on malformed K3D_VERSION before any fetch: the value lands
in a raw.githubusercontent URL path, so a crafted value carrying '/'
could traverse to a script outside k3d-io/k3d and run it privileged.
Accept only vX[.Y.Z…] tags or 'latest'.
- Stop claiming K3D_VERSION="" restores resolve-at-install-time — the
common.sh :- default re-pins on empty (same convention as K8S_VERSION);
only 'latest' opts out. Comments now say so.
- New bats case: malformed tag errors out with nothing fetched/run.
- Manifest regenerated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

bugbot run

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit e69ec70. Configure here.

Comment threadscripts/lib/setup-linux.sh Outdated
shujaat hasanand others added 3 commits July 24, 2026 15:15
… nothing (review)
Asad pulled the pinned v5.9.0 install.sh: downloadFile fetches the bare
binary, installFile chmod+cp's it — no checksum is downloaded or verified,
so the previous comment/PR claim ('the script still verifies the binary')
was wrong, on a sudo path.
- Linux: drop upstream install.sh entirely; fetch k3d-linux-<arch> +
checksums.txt straight from the pinned release and sha256-verify before
installing (mirrors _fetch_kubectl, fail-closed). K3D_VERSION=latest now
resolves the tag via the plain /releases/latest redirect and takes the
SAME verified path.
- Windows: the ps1 fetched sha256sum.txt, which has never existed on k3d
releases (asset is checksums.txt) — it 404'd into a fail-open catch, so
that verification silently never ran either. URL fixed.
- preflight: the k3d tool-host probe now points at github.com (the install
script from raw.githubusercontent.com is no longer fetched).
- bats: 5 tests for the new path (verified download / mismatch fail-closed /
missing-asset fail-closed / sudo target / latest-resolve); the 3
script-era tests removed with the script.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

bugbot run

Comment threadscripts/lib/setup-linux.sh Outdated
- connect-timeout + speed-limit floor on the release downloads (no --max-time:
the ~50 MB binary on a slow-but-healthy link must not be capped), max-time on
the small releases/latest HEAD resolve — a hung transfer under spin_cmd would
otherwise spin forever
- comments: only the literal K3D_VERSION=latest opts out; also corrected the
adjacent pre-existing K8S_VERSION comment that claimed "" resolves latest
(the :- default re-pins on empty)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

bugbot run

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 6b25719. Configure here.

Comment threadscripts/install-k8s.ps1
An unfetchable checksums.txt, a missing asset line, or a mismatch now all
abort and remove the download — matching the Linux path and the kubectl
precedent — instead of skipping silently or logging into a fail-open catch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

bugbot run

saadqbal
saadqbal previously approved these changes Jul 24, 2026
@LukasWodka
LukasWodka merged commit 84e4b18 into developJul 24, 2026
31 checks passed
@shujaatTracebloc
shujaatTracebloc deleted the fix/installer-pin-k3d branch July 24, 2026 14:05
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

Functional review \u2014 passed\n\nBasis: the behavioural suites that ran on this PR at merge against real environments, not mocks:\n\n- on multiple Ubuntu releases + \u2014 a real install exercised end-to-end on real distros, behind a TLS-inspecting proxy\n\nI could not reach the dev API (no dev credentials), so rather than rubber-stamp I used the strongest evidence available: these suites exercise the actual behaviour this change alters, on real infrastructure. Advancing \u2192 .\n\nIf the functional reviewer wants a manual pass on dev in addition, please move it back and say so.

LukasWodka added a commit that referenced this pull request Jul 27, 2026
…load, no get-helm-3 (#396)
* feat(installer): self-serve Helm prerequisites — verified direct download, no get-helm-3 (#395)
The installer now takes care of its own requirements instead of erroring
out and telling the user to install tools:
- install_helm fetches the pinned Helm release directly from get.helm.sh
and verifies it against the published .sha256sum (fail-closed), exactly
like the k3d direct download (#382). helm's get-helm-3 script is gone —
it floats on the mutable helm/helm@main, performs its checksum step with
openssl (absent on minimal cloud images, Bugbot #383), and its fetches
are unbounded. openssl is no longer needed anywhere in the flow.
- HELM_VERSION pin in common.sh (v4.2.3; 'latest' resolves at install
time via get.helm.sh/helm-latest-version, same verified path; malformed
tags fail closed before any fetch).
- _ensure_unpack_tools: when tar/gzip are genuinely missing on the Tier 0
fast path (which skips install_system_deps), install them via the
package manager — quietly as root/passwordless sudo, with an honest
one-line reason when a password is needed — rather than aborting with
'go install tar'. install_system_deps drops openssl, adds gzip.
Supersedes the fatal-error approach in #395 (product call: never tell the
user to install tools we can install ourselves). Windows already fetches
Helm directly from get.helm.sh, so the ps1 path is untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(installer): route sudo probes through _real_sudo/_have_sudo_bin (Bugbot r1)
The A2 sudo shadow runs '-n true' as a command when root, and 'has sudo'
is always true because the shadow function exists — use the #372
primitives like preflight_sudo/_probe_privilege do. The root-without-
sudo-binary strip is dropped entirely: the shadow already handles root by
executing the command directly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(installer): Tier 0 unpack install — sudo keepalive + dpkg-lock wait + one combined install (Bugbot r2)
_ensure_unpack_tools ran package installs on the Tier 0 path without the
full flow's guards: apt could sit on the dpkg lock invisibly behind the
spinner, and a long wait could outlast the just-primed sudo timestamp so
the next sudo re-prompts behind the spinner and hangs. Prime, then keep
the ticket warm (preflight_sudo's pattern; killed right after the install
— the zero-privilege tier shouldn't hold a warm admin ticket), wait out
the dpkg lock (bounded + visible), and install everything in ONE package-
manager call so there's a single sudo consumer right after priming.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(installer): keepalive ownership + latest-tag isolation (Bugbot r3)
- _ensure_unpack_tools kills only the keepalive IT started: on the Tier
1/2 recovery path SUDO_KEEPALIVE_PID belongs to preflight_sudo, and
killing it would let later privileged steps re-prompt behind a spinner.
The global is only claimed when empty (install_cleanup coverage) and
only cleared when it is ours.
- HELM_VERSION=latest: isolate the endpoint body with tail -1 — retry's
attempt notices go to stdout and a failed-then-successful fetch would
concatenate them into the captured tag, failing the anchored regex with
a false 'couldn't resolve'. (install_k3d's resolver is immune: its
${var##*/} strip discards anything before the redirect URL.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(copy-catalog): regenerate 00-install golden for the HELM_VERSION knob
The PR adds a HELM_VERSION env var (common.sh), which surfaces one line in
install-k8s.sh --help. emit_install reads --help live, so the golden drifted
by exactly that line. Regenerated via TB_UPDATE_GOLDEN=1; verified locally
(bats copy-catalog now green).
* fix(installer): drop obsolete _ensure_helm_prereqs — Tier 0 no longer blocks on openssl (Bugbot #396)
get-helm-3 is gone (replaced by a sha256-verified direct download), so Helm
no longer needs openssl. The Tier-0 preflight still demanded openssl+tar and
failed fast, aborting minimal-image installs for a dependency Helm doesn't use
— and pre-empting _ensure_unpack_tools, which installs tar/gzip Tier-0-aware.
Remove the function and its Tier-0 call; drop the tests that encoded the old
openssl preflight (tar/gzip stays covered by _ensure_unpack_tools' own tests).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
LukasWodka added a commit that referenced this pull request Jul 27, 2026
…lookups (#438)
* fix(installer): pin k3d + helm versions — drop api.github.com latest lookups (#410)
Ports the bash pins (#382) to the Windows installer: K3D_VERSION defaults
to v5.9.0 and HELM_VERSION to v4.2.3 (lockstep with scripts/lib/common.sh
until #435 single-sources them), env-overridable, validated against a
release-tag shape before any URL is built (path-traversal gate mirroring
the bootstrap and cli install.ps1).
The unauthenticated releases/latest API allows 60 req/hour per IP — one
shared corporate NAT exhausts it and fails installs (observed live on a
customer install 2026-07-27). The literal value 'latest' still works but
resolves API-free: the /releases/latest redirect Location for k3d, and
get.helm.sh/helm-latest-version for helm — exactly like lib/setup-linux.sh.
A Pester gate now fails the suite if any https://api.github.com fetch
reappears in the installer.
Closes#410
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(installer): restore retry parity on the 'latest' version resolvers (Bugbot #438)
The new API-free 'latest' resolvers ran a single request — the old
lookups and lib/setup-linux.sh retry 3x5s. Resolve-ToolVersion now
drives the resolver through Invoke-WithRetry (resolvers throw on
failure), so a one-off network blip on flaky corporate egress retries
instead of aborting the install; a persistent failure still fails
closed with the pin-a-tag remedy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(installer): bound the 'latest' lookup requests (-TimeoutSec 30, Bugbot #438)
A host that accepts the TCP connect but never responds would hang the
version resolvers indefinitely; the bash peers bound this with
--connect-timeout 15 --max-time 30. Both resolver requests now carry
-TimeoutSec 30; a timeout throws, so the retry ladder + fail-closed
remedy from the previous commit take over. AST-based Pester gate keeps
the timeouts in place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
shujaatTracebloc added a commit that referenced this pull request Jul 27, 2026
…lookups (re-land #410 onto develop) (#446)
* fix(installer): pin k3d + helm versions — drop api.github.com latest lookups (#410)
Ports the bash pins (#382) to the Windows installer: K3D_VERSION defaults
to v5.9.0 and HELM_VERSION to v4.2.3 (lockstep with scripts/lib/common.sh
until #435 single-sources them), env-overridable, validated against a
release-tag shape before any URL is built (path-traversal gate mirroring
the bootstrap and cli install.ps1).
The unauthenticated releases/latest API allows 60 req/hour per IP — one
shared corporate NAT exhausts it and fails installs (observed live on a
customer install 2026-07-27). The literal value 'latest' still works but
resolves API-free: the /releases/latest redirect Location for k3d, and
get.helm.sh/helm-latest-version for helm — exactly like lib/setup-linux.sh.
A Pester gate now fails the suite if any https://api.github.com fetch
reappears in the installer.
Closes#410
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(installer): restore retry parity on the 'latest' version resolvers (Bugbot #438)
The new API-free 'latest' resolvers ran a single request — the old
lookups and lib/setup-linux.sh retry 3x5s. Resolve-ToolVersion now
drives the resolver through Invoke-WithRetry (resolvers throw on
failure), so a one-off network blip on flaky corporate egress retries
instead of aborting the install; a persistent failure still fails
closed with the pin-a-tag remedy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(installer): bound the 'latest' lookup requests (-TimeoutSec 30, Bugbot #438)
A host that accepts the TCP connect but never responds would hang the
version resolvers indefinitely; the bash peers bound this with
--connect-timeout 15 --max-time 30. Both resolver requests now carry
-TimeoutSec 30; a timeout throws, so the retry ladder + fail-closed
remedy from the previous commit take over. AST-based Pester gate keeps
the timeouts in place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Lukas Wuttke <lukas@tracebloc.io>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
shujaatTracebloc added a commit that referenced this pull request Aug 3, 2026
…RFC D3/D4) (#528)
* feat(#435): single-source cross-OS installer facts + CI parity gate (RFC D3/D4)
The costliest drift class of the installer sweep was FACTS diverging between the three
OS implementations — the #410 incident (k3d/helm pins bumped in bash #382 but not
PowerShell #410) failed a real customer install. Copy already had the byte-exact catalog;
behavior facts get the same treatment here.
- scripts/spec/facts.env: the single source of truth for cross-OS facts. Tool version
pins (K3D_VERSION / HELM_VERSION / K8S_VERSION) + the READY_TIMEOUT budget.
- scripts/check-facts.sh: --write stamps the spec into every consumer (bash common.sh +
summary.sh, PowerShell install-k8s.ps1); --check is the CI gate (mirrors gen-manifest's
write/check split). Nothing is sourced at runtime — consumers carry literal values, so
the single-file verified bootstrap (R8) is untouched.
- CI: installer-tests.yaml runs `check-facts.sh --check`, failing the PR if any consumer
drifted from the spec — so the #410 incident (a pin in one OS path but not the other)
can no longer ship. install-k8s.ps1's lockstep comment updated to point at the spec.
Tests: scripts/tests/check-facts.bats — the #410 incident reproduced as a red check
(both directions), --write round-trips for versions + the timeout, fail-closed on a
missing pattern, bad-mode rejection.
Scope note: hosts are already single-sourced + drift-checked (check-drift Checks 1 & 5);
memory floors (bash-only today) and the behavior-parity matrix are follow-ups on this
same mechanism — a new fact is one row in facts.env + check-facts.sh.
Closes#435
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(#435): enforce K8S_VERSION in PowerShell too + shellcheck check-facts.sh (Bugbot)
Two Bugbot findings:
1. (High) install-k8s.ps1 ALSO pins K8S_VERSION ($K8S_VERSION default, passed to k3d as
--image rancher/k3s:$K8S_VERSION), but check-facts treated the pin as bash-only — so
bumping the spec + --write updated only common.sh, Windows stayed stale, and --check
stayed green: the exact #410 hole for this pin. Added install-k8s.ps1 as a K8S_VERSION
consumer (extract + rewrite), updated the facts.env note.
2. (Low) scripts/check-facts.sh was wired for --check but not in the explicit
shellcheck --severity=error file list (unlike gen-manifest.sh) — a regression in the
facts gate wouldn't fail static CI. Added it to both shellcheck lines.
Tests: K8S_VERSION drift in PowerShell -> RED; a K8S bump stamps BOTH consumers. 11/11.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@LukasWodka@saadqbal@shujaatTracebloc