Skip to content

fix(installer): survive a proxy/AV-truncated tool download (k3d/kubectl/helm) (#607) - #608

Merged
shujaatTracebloc merged 4 commits into
developfrom
fix/607-resilient-tool-download
Aug 5, 2026
Merged

fix(installer): survive a proxy/AV-truncated tool download (k3d/kubectl/helm) (#607)#608
shujaatTracebloc merged 4 commits into
developfrom
fix/607-resilient-tool-download

Conversation

@shujaatTracebloc

@shujaatTraceblocshujaatTracebloc commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What & why

Any user installing on a filtered corporate network can dead-end here — via the official one-liner, not just a dev build. A proxy or antivirus can truncate or block a GitHub-release binary mid-transfer; install-k8s.ps1 had a single transport (Invoke-WebRequest) and a single after-the-fact signal (the checksum), so a blocked k3d download stopped the install at the cryptic System tool checksum verification failed. This is the field failure behind #578 ("installer must complete on restricted/corporate networks").

This is not specific to how the installer is launched. The kubectl/k3d/helm downloads happen in Step 2 ("Install system tools") of install-k8s.ps1, which runs identically no matter the entry point — the official releases/download/<tag>/install.ps1 one-liner, a .tgz, or a dev branch. Step 2 always fetches the tools from the same upstream URLs (github.com/k3d-io/…, dl.k8s.io, get.helm.sh) over the user's own network, so the same proxy/AV that truncates the transfer fails the install regardless of source. (It happened to surface during testing, but the install method was not the cause — a real user on that network hits it from the official link.)

Also: the k3d winget fallback was dead weight — k3d has no winget manifest, so Rancher.k3d always returned "No package found" before the direct download ran anyway.

Closes#607. Contributes to #578.

PowerShell (install-k8s.ps1)

  • Get-VerifiedDownload tries three transports in turn — Invoke-WebRequestcurl.exeBITS. A different HTTP stack commonly succeeds where one is blocked/truncated (the primary fix).
  • Test-DownloadComplete validates each result before the checksum: present, ≥ a per-tool size floor, and the expected magic bytes (MZ for a .exe, PK for the helm zip). A short / error-page / altered payload is caught as a transfer failure (distinct from a checksum mismatch) and the next transport is tried.
  • kubectl / k3d / helm all route through it.
  • Removed the dead k3d winget branch (verified k3d has no winget manifest — every id variant 404s).
  • Every-transport-failure throws one specific, actionable message (allowlist the hosts / exclude the tools dir from AV), not the cryptic checksum error.

Bash parity (lib/common.sh, lib/setup-linux.sh)

  • _assert_download_size gates kubectl/k3d/helm on a size floor before the checksum, so a truncated/blocked transfer reports the real reason. (Linux already uses curl.) TB_MIN_DOWNLOAD_BYTES lets the bats fetch mocks — tiny fixture files — relax the floor; the real floor applies in production.

Tests

  • Pester:Test-DownloadComplete (7 cases: complete .exe/.zip, truncated, too-small, wrong-magic, missing, magic-skipped) + a guard that the k3d winget branch is gone. Full Pester suite green (436).
  • bats:_assert_download_size (4 cases) + the env hook; existing fetch mocks updated to set TB_MIN_DOWNLOAD_BYTES=0. setup-linux / setup-macos / common / check-drift all green; assertion-enforce clean.
  • Manifest regenerated (install-k8s.ps1 / common.sh / setup-linux.sh are manifested).

For reference: the field symptom (official-link install on a corporate machine)

Installing k3d (winget): No package found matching input criteria
Downloading k3d binary directly (amd64)... (14:04:04)
ERROR: System tool checksum verification failed (14:04:07 — 3s for a 28 MB binary)

Not auto-fixed (by design)

If AV delivers a complete but rewritten binary (full size, valid magic, altered bytes), the checksum still fails — correctly — and the user gets the clear "proxy/AV may be rewriting the binary → add an exclusion" message. Antivirus-specific auto-exclusion and an internal tools-mirror override were deliberately left out of scope.

🤖 Generated with Claude Code


Note

Low Risk
Installer-only download/validation changes with parity tests; no runtime cluster or auth behavior changes.

Overview
Windows (install-k8s.ps1) adds Test-DownloadComplete (size + optional MZ/PK magic) and Get-VerifiedDownload, which retries kubectl/k3d/helm fetches across Invoke-WebRequest → curl.exe → BITS, only accepting complete payloads before checksum. Failed transfers surface an allowlist/AV message instead of a generic checksum error. The dead k3d winget path is removed (no manifest).

Linux/macOS adds _assert_download_size in common.sh and calls it before checksum verification for kubectl, k3d, and Helm in setup-linux.sh, with the same transfer-vs-tampering distinction. TB_MIN_DOWNLOAD_BYTES=0 in bats mocks keeps existing fetch tests working.

Tests cover the new helpers; manifest.sha256 is updated for touched scripts.

Reviewed by Cursor Bugbot for commit 1410766. Bugbot is set up for automated code reviews on this repo. Configure here.

…tl/helm) (#607)
On a filtered corporate network a proxy or antivirus can truncate or block a
GitHub-release binary mid-transfer. install-k8s.ps1 had a single transport
(Invoke-WebRequest) and a single after-the-fact signal (the checksum), so a
blocked k3d download dead-ended at the cryptic "System tool checksum
verification failed" -- the #578 field failure. The winget fallback was also
dead (k3d has no winget manifest, so Rancher.k3d always returned "No package
found").
PowerShell (install-k8s.ps1):
- Get-VerifiedDownload tries three transports in turn -- Invoke-WebRequest, then
curl.exe, then BITS. A different HTTP stack commonly succeeds where one is
blocked/truncated.
- Test-DownloadComplete validates each result BEFORE the checksum: present, at
least a per-tool size floor, and the expected magic bytes (MZ for .exe, PK for
the helm zip). A short/error-page/altered payload is caught as a TRANSFER
failure (distinct from a checksum mismatch) and the next transport is tried.
- kubectl / k3d / helm all route through it.
- Removed the dead k3d winget branch (verified: k3d has no winget manifest).
- Every-transport-failure throws one specific, actionable message (allowlist the
hosts / exclude the tools dir from AV) instead of the cryptic checksum error.
Bash parity (lib/common.sh, lib/setup-linux.sh):
- _assert_download_size gates kubectl/k3d/helm downloads on a size floor before
the checksum, so a truncated/blocked transfer reports the real reason. The
Linux path already uses curl; TB_MIN_DOWNLOAD_BYTES lets the bats fetch mocks
(tiny fixtures) relax the floor.
Tests: Pester (Test-DownloadComplete, 7) + winget-removal guard; bats
(_assert_download_size, 4); fetch mocks updated via the env hook. Manifest
regenerated.
Contributes to #578. Closes#607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@shujaatTraceblocshujaatTracebloc self-assigned this Aug 5, 2026
@shujaatTracebloc
shujaatTracebloc marked this pull request as ready for review August 5, 2026 12:54
… style guard
curl_secure() is a bash helper and cannot exist in PowerShell, so the resilient
download's curl.exe fallback is a deliberate, flag-matched fetch. Mark both lines
(the transport + the presence check) with '# style-guard: allow'. Regenerates
the manifest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment threadscripts/install-k8s.ps1 Outdated
Comment threadscripts/install-k8s.ps1
…abort the fallback loop (Bugbot)
- curl.exe fallback now passes --tlsv1.2, matching curl_secure's TLS floor, so it
cannot negotiate below TLS 1.2 on the proxy networks this targets.
- Get-VerifiedDownload wraps the Test-DownloadComplete call in try/catch: a
post-download I/O error (e.g. AV locking/quarantining the just-written file) now
records a problem and tries the next transport instead of aborting the whole
download — the recovery path is the point of this change.
- Pester source-guards for both; manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8b4fc71. Configure here.

Comment threadscripts/lib/common.sh
_assert_download_size calls error (which exits); it now removes the caller's
mktemp -d tree first (passed as $4) so a truncated/blocked transfer no longer
leaves a partial tool payload under /tmp — matching the checksum-mismatch
branches in _fetch_kubectl / _fetch_k3d_release / _fetch_helm_release. Adds bats
coverage for both the cleanup-on-failure and leave-intact-on-success paths;
manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@shujaatTracebloc
shujaatTracebloc merged commit 6d8760a into developAug 5, 2026
47 checks passed
@shujaatTracebloc
shujaatTracebloc deleted the fix/607-resilient-tool-download branch August 5, 2026 13:38
shujaatTracebloc added a commit that referenced this pull request Aug 5, 2026
…ated tool binary self-heals (#611)
Field follow-up to #607/#608. On a Windows machine behind a filtering proxy, the
k3d download kept failing at "System tool checksum verification failed" even with
#608's multi-transport download — because #608 validated a download by SIZE FLOOR
+ magic bytes only. A binary truncated mid-transfer to somewhere between the 10 MB
floor and the real 25.8 MB still passed (it's >10 MB and starts with 'MZ'), so the
fallbacks never fired and it dead-ended at the separate, no-retry checksum step.
(Proven on the box: a manual download produced the correct hash at 25,805,312
bytes, while the installer's copy failed the checksum in ~2s with no fallback.)
Fix — the checksum is the authoritative completeness test:
- Get-VerifiedDownload gains -Sha256: after a transport lands a size/magic-valid
file, its SHA-256 must equal the expected hash or the transport is treated as
failed and the NEXT one (curl.exe -> BITS) is tried. A truncated/altered copy
now self-heals instead of dead-ending.
- Get-VerifiedDownload gains -MustContain for the small checksum-list files, so a
proxy error page lacking the expected asset line is retried too.
- k3d / kubectl / helm now fetch their checksum FIRST (resiliently) and pass the
extracted, 64-hex-validated hash as the download gate. helm gains checksum
verification on the PS path for the first time (parity with the bash path).
Tests: Pester source-guards for the -Sha256/-MustContain gates, the mismatch->retry
path, and per-tool checksum-first wiring; full suite green (444). Manifest
regenerated.
Contributes to #578. Closes#611.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
shujaatTracebloc added a commit that referenced this pull request Aug 6, 2026
…ecksum-driven downloads, reliable cluster-create, writable ingest & training volumes, local-chart support (#611) (#612)
* fix(installer): make the checksum drive the download retry so a truncated tool binary self-heals (#611)
Field follow-up to #607/#608. On a Windows machine behind a filtering proxy, the
k3d download kept failing at "System tool checksum verification failed" even with
#608's multi-transport download — because #608 validated a download by SIZE FLOOR
+ magic bytes only. A binary truncated mid-transfer to somewhere between the 10 MB
floor and the real 25.8 MB still passed (it's >10 MB and starts with 'MZ'), so the
fallbacks never fired and it dead-ended at the separate, no-retry checksum step.
(Proven on the box: a manual download produced the correct hash at 25,805,312
bytes, while the installer's copy failed the checksum in ~2s with no fallback.)
Fix — the checksum is the authoritative completeness test:
- Get-VerifiedDownload gains -Sha256: after a transport lands a size/magic-valid
file, its SHA-256 must equal the expected hash or the transport is treated as
failed and the NEXT one (curl.exe -> BITS) is tried. A truncated/altered copy
now self-heals instead of dead-ending.
- Get-VerifiedDownload gains -MustContain for the small checksum-list files, so a
proxy error page lacking the expected asset line is retried too.
- k3d / kubectl / helm now fetch their checksum FIRST (resiliently) and pass the
extracted, 64-hex-validated hash as the download gate. helm gains checksum
verification on the PS path for the first time (parity with the bash path).
Tests: Pester source-guards for the -Sha256/-MustContain gates, the mismatch->retry
path, and per-tool checksum-first wiring; full suite green (444). Manifest
regenerated.
Contributes to #578. Closes#611.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(#611): content-gate the kubectl .sha256 fetch so a proxy page retries transports (Bugbot)
The kubectl .sha256 is a bare 64-hex hash with no fixed substring, so it used
-MinBytes 1 with no content gate -- a proxy error page satisfied the floor, the
first transport 'succeeded', curl.exe/BITS never ran, and the later hex check
aborted. Add -MatchPattern (a regex content gate) to Get-VerifiedDownload and use
'[0-9a-fA-F]{64}' for the kubectl checksum fetch, matching how k3d/helm use
-MustContain. Pester guards updated; manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(installer): don't misread a successful k3d cluster-create as failed (#611)
Field report (same Windows box, past the k3d download fix): Step 3 aborted with
"Failed to create compute environment" even though k3d printed
"Cluster 'tracebloc' created successfully!" with EMPTY stderr and the cluster was
actually up. Cause: Wait-ProcessWithDeadline polled HasExited but never called
WaitForExit(), and Start-Process -RedirectStandardOutput can leave $proc.ExitCode
$null in that window -- so `$null -ne 0` misread an exit-0 success as a failure.
Not machine-specific; a latent race any Windows user can hit.
- Wait-ProcessWithDeadline now calls $Process.WaitForExit() before returning
success, so the redirected streams drain and ExitCode is reliable for EVERY
caller (cluster create, partial delete, tracked installs).
- Cluster-create adds defense-in-depth: a still-null exit code falls back to
k3d's own "created successfully" marker rather than failing a cluster that is up.
- Pester source-guards for both. Windows-only change (install-k8s.ps1); Linux/mac
paths untouched and their suites remain green.
Contributes to #578.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(#611): hash-anchor the checksum-list gates + check both k3d streams in the exit fallback (Bugbot)
Two Bugbot findings on the checksum-driven download work:
1. (High) The checksum-LIST fetch gates were fail-open, so a proxy error page
"succeeded" on the first transport and skipped the curl.exe/BITS retry — exactly
the case #611 exists to survive. Helm's -MustContain substring
(helm-<ver>-windows-<arch>.zip) also appears in the request URL a proxy page can
echo; kubectl's -MatchPattern was unanchored so any page with a 64-hex run passed;
k3d gated on the bare asset name. Fix: drop the weak -MustContain entirely and gate
every checksum-list fetch on the hash STRUCTURE — k3d/helm require a 64-hex hash
adjacent to the asset, kubectl requires the hash at the start of the body. A
proxy/HTML error page can't satisfy that, so it retries transports as intended.
2. (Medium) The null-exit-code cluster-create fallback only scanned $k3dStdout, but
k3d logs its "Cluster created successfully!" line via logrus to STDERR — so a real
success could be misread as failure. Fix: check both $k3dStdout and $k3dStderr.
Pester source-guards updated: kubectl gate is start-anchored, k3d/helm gates are
hash-anchored, no -MustContain remains, and the fallback inspects both streams. Full
suite green (447). Manifest regenerated.
Contributes to #578.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(#611): make /data/shared writable so dataset ingest works on hostPath installs
`tb data ingest` failed at the copy step with `mkdir: can't create directory
'/data/shared/.tracebloc-staging/': Permission denied`. On hostPath installs (the
Windows/WSL2 + bare-metal default) kubelet does not apply fsGroup to hostPath
volumes (kubernetes/kubernetes#138411), so /data/shared (client-pvc) is created
root-owned and the non-root ingest-staging pod can't write to it. mysql-data has a
privileged init-chown for exactly this reason; the shared data volume had none.
- jobs-manager gains fsGroup: 1000 (CSI clusters apply it to the shared volume).
- On hostPath, a privileged init-shared-data container (root, CHOWN+FOWNER only)
chowns /data/shared to 1000:1000 and chmod 2777. World-writable, unlike
mysql-data's single-UID chown, because the shared volume has multiple non-root
writers whose UIDs this chart doesn't control -- jobs-manager, the training/
ingestor pods it spawns, and the CLI's ingest-staging pod. setgid keeps new files
in GID 1000; the init is gated on hostPath (CSI relies on fsGroup).
- helm-unittest: init present + world-writable on hostPath; absent (fsGroup kept)
on CSI. Chart bumped 1.9.15 -> 1.9.16.
Client-side companion to the installer fixes on this PR (requested to land here).
Contributes to #578.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(installer): support TRACEBLOC_CHART_PATH on Windows (local-chart parity with bash)
The Windows installer could only ever install the PUBLISHED chart (helm repo), so a
branch-only chart change (e.g. the #611 /data/shared fix, or #585's global.imageRegistry)
was impossible to test from a Windows install. The bash installer already supports a
local chart via TRACEBLOC_CHART_PATH (_resolve_chart_ref); this brings Windows to parity.
- When $env:TRACEBLOC_CHART_PATH is set, install-k8s.ps1 installs from that local chart
directory (validated) and skips `helm repo add`; otherwise it uses the published repo
as before. Applied to both the fresh-install and adopt/reconcile helm upgrades.
- Pester source-guards for the local-chart ref, the repo-add skip, and the not-a-directory
error. Manifest regenerated. Full suite green (451).
Enables a from-scratch Windows test of the branch chart. Contributes to #578.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(installer): make /data/logs writable for training & inference pods (#611)
hostPath ignores fsGroup (kubernetes/kubernetes#138411), so /data/logs was
created root-owned and non-root training/inference pods hit
`PermissionError [Errno 13]` creating their per-run log dir
(`os.makedirs('/data/logs/<run>')`). The #611 init-container chowned
/data/shared but not /data/logs — the same class of bug on the logs volume.
Extend the init (renamed init-shared-data -> init-writable-data) to
chown+chmod BOTH hostPath volumes and mount both. Training and inference
pods share one spec (job.yaml), so this covers both; ingestion already
covered by the /data/shared chmod. Bump chart 1.9.16 -> 1.9.17.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(installer): address review — sticky bit on shared dirs + reset-then-reuse parity
Two reviewer follow-ups on #612:
- chart: chmod the writable hostPath dirs 3777 (was 2777) — add the sticky
bit so one writer can't unlink/rename another writer's files in
/data/shared // /data/logs (/tmp semantics). setgid is retained. Safe given
the uid topology (dir owned by 1000; training pods run as 1000; the ingestor
writes its own subtrees as a stable uid) and no cross-uid filesystem deletes
exist in client-runtime. Chart 1.9.17 -> 1.9.18.
- install-k8s.ps1: the adopt/reconcile helm upgrade now prefers
--reset-then-reuse-values when `helm upgrade --help` advertises it (Helm
>= 3.14), falling back to --reuse-values otherwise — so NEW chart defaults
reach adopted Windows edges on auto-upgrade (bash parity with
install-client-helm.sh). manifest.sha256 regenerated.
Tests: helm-unittest updated (3777); new Pester test asserts the
reset-then-reuse preference; full Pester suite 452 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(installer): add CAP_FSETID so the init's setgid bit survives (Bugbot)
init-writable-data chowns the shared/logs hostPath dirs to GID 1000 then
chmods 3777. With caps dropped to CHOWN+FOWNER only, the kernel silently
strips S_ISGID on the chmod — after the chown the dir's group no longer
matches the process (fsgid 0), and a root process without CAP_FSETID can't
set setgid on it — so the mount landed at 1777 and new files did NOT inherit
GID 1000 as documented. Add FSETID to the cap set; setgid now sticks.
helm-unittest asserts FSETID present. Chart 1.9.18 -> 1.9.19.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(installer): drop risky fsGroup + make init per-volume best-effort (Bugbot)
Two Bugbot findings on the writable-volume fix:
- HIGH — remove `fsGroup: 1000` / `fsGroupChangePolicy` from jobs-manager. It is
a no-op on hostPath (kubelet ignores fsGroup — the init does the work) and on
CSI it only grants jobs-manager's OWN processes GID 1000 while its
OnRootMismatch relabel flips the shared/logs volumes to group 1000 — stripping
the group-0 access the spawned training pods (UID 1001 / OpenShift arbitrary
UID, GID 0) and the host-UID ingestion pods rely on (docs/SECURITY.md §5.3). It
never reaches those spawned writers, so it was all regression risk and no gain.
Those pods keep their own documented posture; CSI is untouched (matches develop).
- MEDIUM — the init now fixes each dir INDEPENDENTLY and best-effort:
`for d in /data/shared /data/logs; do chown && chmod || echo <warn>; done`. A
chown that can't complete (e.g. /data/shared on an NFS root_squash export) no
longer aborts the chain and skips /data/logs — the other dir is still repaired
and jobs-manager still starts; a truly unwritable mount surfaces as a clear
error at the writer pod instead of wedging the edge in Init.
helm-unittest updated (no fsGroup on either path; per-dir loop; CSI skips init).
Chart 1.9.19 -> 1.9.20.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Installer tool acquisition must survive a proxy/AV-truncated binary download (k3d/kubectl/helm)

3 participants

@shujaatTracebloc@aptracebloc@LukasWodka