Uh oh!
There was an error while loading. Please reload this page.
fix(installer): survive a proxy/AV-truncated tool download (k3d/kubectl/helm) (#607) - #608
Merged
Merged
Conversation
…tl/helm) (#607) On a filtered corporate network a proxy or antivirus can truncate or block a GitHub-release binary mid-transfer. install-k8s.ps1 had a single transport (Invoke-WebRequest) and a single after-the-fact signal (the checksum), so a blocked k3d download dead-ended at the cryptic "System tool checksum verification failed" -- the #578 field failure. The winget fallback was also dead (k3d has no winget manifest, so Rancher.k3d always returned "No package found"). PowerShell (install-k8s.ps1): - Get-VerifiedDownload tries three transports in turn -- Invoke-WebRequest, then curl.exe, then BITS. A different HTTP stack commonly succeeds where one is blocked/truncated. - Test-DownloadComplete validates each result BEFORE the checksum: present, at least a per-tool size floor, and the expected magic bytes (MZ for .exe, PK for the helm zip). A short/error-page/altered payload is caught as a TRANSFER failure (distinct from a checksum mismatch) and the next transport is tried. - kubectl / k3d / helm all route through it. - Removed the dead k3d winget branch (verified: k3d has no winget manifest). - Every-transport-failure throws one specific, actionable message (allowlist the hosts / exclude the tools dir from AV) instead of the cryptic checksum error. Bash parity (lib/common.sh, lib/setup-linux.sh): - _assert_download_size gates kubectl/k3d/helm downloads on a size floor before the checksum, so a truncated/blocked transfer reports the real reason. The Linux path already uses curl; TB_MIN_DOWNLOAD_BYTES lets the bats fetch mocks (tiny fixtures) relax the floor. Tests: Pester (Test-DownloadComplete, 7) + winget-removal guard; bats (_assert_download_size, 4); fetch mocks updated via the env hook. Manifest regenerated. Contributes to #578. Closes#607. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… style guard curl_secure() is a bash helper and cannot exist in PowerShell, so the resilient download's curl.exe fallback is a deliberate, flag-matched fetch. Mark both lines (the transport + the presence check) with '# style-guard: allow'. Regenerates the manifest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…abort the fallback loop (Bugbot) - curl.exe fallback now passes --tlsv1.2, matching curl_secure's TLS floor, so it cannot negotiate below TLS 1.2 on the proxy networks this targets. - Get-VerifiedDownload wraps the Test-DownloadComplete call in try/catch: a post-download I/O error (e.g. AV locking/quarantining the just-written file) now records a problem and tries the next transport instead of aborting the whole download — the recovery path is the point of this change. - Pester source-guards for both; manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8b4fc71. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
_assert_download_size calls error (which exits); it now removes the caller's mktemp -d tree first (passed as $4) so a truncated/blocked transfer no longer leaves a partial tool payload under /tmp — matching the checksum-mismatch branches in _fetch_kubectl / _fetch_k3d_release / _fetch_helm_release. Adds bats coverage for both the cleanup-on-failure and leave-intact-on-success paths; manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
aptracebloc
approved these changes
Aug 5, 2026
Uh oh!
There was an error while loading. Please reload this page.
This was referenced Aug 5, 2026
shujaatTracebloc added a commit
that referenced
this pull request
Aug 5, 2026
…ated tool binary self-heals (#611) Field follow-up to #607/#608. On a Windows machine behind a filtering proxy, the k3d download kept failing at "System tool checksum verification failed" even with #608's multi-transport download — because #608 validated a download by SIZE FLOOR + magic bytes only. A binary truncated mid-transfer to somewhere between the 10 MB floor and the real 25.8 MB still passed (it's >10 MB and starts with 'MZ'), so the fallbacks never fired and it dead-ended at the separate, no-retry checksum step. (Proven on the box: a manual download produced the correct hash at 25,805,312 bytes, while the installer's copy failed the checksum in ~2s with no fallback.) Fix — the checksum is the authoritative completeness test: - Get-VerifiedDownload gains -Sha256: after a transport lands a size/magic-valid file, its SHA-256 must equal the expected hash or the transport is treated as failed and the NEXT one (curl.exe -> BITS) is tried. A truncated/altered copy now self-heals instead of dead-ending. - Get-VerifiedDownload gains -MustContain for the small checksum-list files, so a proxy error page lacking the expected asset line is retried too. - k3d / kubectl / helm now fetch their checksum FIRST (resiliently) and pass the extracted, 64-hex-validated hash as the download gate. helm gains checksum verification on the PS path for the first time (parity with the bash path). Tests: Pester source-guards for the -Sha256/-MustContain gates, the mismatch->retry path, and per-tool checksum-first wiring; full suite green (444). Manifest regenerated. Contributes to #578. Closes#611. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
shujaatTracebloc added a commit
that referenced
this pull request
Aug 6, 2026
…ecksum-driven downloads, reliable cluster-create, writable ingest & training volumes, local-chart support (#611) (#612) * fix(installer): make the checksum drive the download retry so a truncated tool binary self-heals (#611) Field follow-up to #607/#608. On a Windows machine behind a filtering proxy, the k3d download kept failing at "System tool checksum verification failed" even with #608's multi-transport download — because #608 validated a download by SIZE FLOOR + magic bytes only. A binary truncated mid-transfer to somewhere between the 10 MB floor and the real 25.8 MB still passed (it's >10 MB and starts with 'MZ'), so the fallbacks never fired and it dead-ended at the separate, no-retry checksum step. (Proven on the box: a manual download produced the correct hash at 25,805,312 bytes, while the installer's copy failed the checksum in ~2s with no fallback.) Fix — the checksum is the authoritative completeness test: - Get-VerifiedDownload gains -Sha256: after a transport lands a size/magic-valid file, its SHA-256 must equal the expected hash or the transport is treated as failed and the NEXT one (curl.exe -> BITS) is tried. A truncated/altered copy now self-heals instead of dead-ending. - Get-VerifiedDownload gains -MustContain for the small checksum-list files, so a proxy error page lacking the expected asset line is retried too. - k3d / kubectl / helm now fetch their checksum FIRST (resiliently) and pass the extracted, 64-hex-validated hash as the download gate. helm gains checksum verification on the PS path for the first time (parity with the bash path). Tests: Pester source-guards for the -Sha256/-MustContain gates, the mismatch->retry path, and per-tool checksum-first wiring; full suite green (444). Manifest regenerated. Contributes to #578. Closes#611. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#611): content-gate the kubectl .sha256 fetch so a proxy page retries transports (Bugbot) The kubectl .sha256 is a bare 64-hex hash with no fixed substring, so it used -MinBytes 1 with no content gate -- a proxy error page satisfied the floor, the first transport 'succeeded', curl.exe/BITS never ran, and the later hex check aborted. Add -MatchPattern (a regex content gate) to Get-VerifiedDownload and use '[0-9a-fA-F]{64}' for the kubectl checksum fetch, matching how k3d/helm use -MustContain. Pester guards updated; manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): don't misread a successful k3d cluster-create as failed (#611) Field report (same Windows box, past the k3d download fix): Step 3 aborted with "Failed to create compute environment" even though k3d printed "Cluster 'tracebloc' created successfully!" with EMPTY stderr and the cluster was actually up. Cause: Wait-ProcessWithDeadline polled HasExited but never called WaitForExit(), and Start-Process -RedirectStandardOutput can leave $proc.ExitCode $null in that window -- so `$null -ne 0` misread an exit-0 success as a failure. Not machine-specific; a latent race any Windows user can hit. - Wait-ProcessWithDeadline now calls $Process.WaitForExit() before returning success, so the redirected streams drain and ExitCode is reliable for EVERY caller (cluster create, partial delete, tracked installs). - Cluster-create adds defense-in-depth: a still-null exit code falls back to k3d's own "created successfully" marker rather than failing a cluster that is up. - Pester source-guards for both. Windows-only change (install-k8s.ps1); Linux/mac paths untouched and their suites remain green. Contributes to #578. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#611): hash-anchor the checksum-list gates + check both k3d streams in the exit fallback (Bugbot) Two Bugbot findings on the checksum-driven download work: 1. (High) The checksum-LIST fetch gates were fail-open, so a proxy error page "succeeded" on the first transport and skipped the curl.exe/BITS retry — exactly the case #611 exists to survive. Helm's -MustContain substring (helm-<ver>-windows-<arch>.zip) also appears in the request URL a proxy page can echo; kubectl's -MatchPattern was unanchored so any page with a 64-hex run passed; k3d gated on the bare asset name. Fix: drop the weak -MustContain entirely and gate every checksum-list fetch on the hash STRUCTURE — k3d/helm require a 64-hex hash adjacent to the asset, kubectl requires the hash at the start of the body. A proxy/HTML error page can't satisfy that, so it retries transports as intended. 2. (Medium) The null-exit-code cluster-create fallback only scanned $k3dStdout, but k3d logs its "Cluster created successfully!" line via logrus to STDERR — so a real success could be misread as failure. Fix: check both $k3dStdout and $k3dStderr. Pester source-guards updated: kubectl gate is start-anchored, k3d/helm gates are hash-anchored, no -MustContain remains, and the fallback inspects both streams. Full suite green (447). Manifest regenerated. Contributes to #578. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#611): make /data/shared writable so dataset ingest works on hostPath installs `tb data ingest` failed at the copy step with `mkdir: can't create directory '/data/shared/.tracebloc-staging/': Permission denied`. On hostPath installs (the Windows/WSL2 + bare-metal default) kubelet does not apply fsGroup to hostPath volumes (kubernetes/kubernetes#138411), so /data/shared (client-pvc) is created root-owned and the non-root ingest-staging pod can't write to it. mysql-data has a privileged init-chown for exactly this reason; the shared data volume had none. - jobs-manager gains fsGroup: 1000 (CSI clusters apply it to the shared volume). - On hostPath, a privileged init-shared-data container (root, CHOWN+FOWNER only) chowns /data/shared to 1000:1000 and chmod 2777. World-writable, unlike mysql-data's single-UID chown, because the shared volume has multiple non-root writers whose UIDs this chart doesn't control -- jobs-manager, the training/ ingestor pods it spawns, and the CLI's ingest-staging pod. setgid keeps new files in GID 1000; the init is gated on hostPath (CSI relies on fsGroup). - helm-unittest: init present + world-writable on hostPath; absent (fsGroup kept) on CSI. Chart bumped 1.9.15 -> 1.9.16. Client-side companion to the installer fixes on this PR (requested to land here). Contributes to #578. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): support TRACEBLOC_CHART_PATH on Windows (local-chart parity with bash) The Windows installer could only ever install the PUBLISHED chart (helm repo), so a branch-only chart change (e.g. the #611 /data/shared fix, or #585's global.imageRegistry) was impossible to test from a Windows install. The bash installer already supports a local chart via TRACEBLOC_CHART_PATH (_resolve_chart_ref); this brings Windows to parity. - When $env:TRACEBLOC_CHART_PATH is set, install-k8s.ps1 installs from that local chart directory (validated) and skips `helm repo add`; otherwise it uses the published repo as before. Applied to both the fresh-install and adopt/reconcile helm upgrades. - Pester source-guards for the local-chart ref, the repo-add skip, and the not-a-directory error. Manifest regenerated. Full suite green (451). Enables a from-scratch Windows test of the branch chart. Contributes to #578. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): make /data/logs writable for training & inference pods (#611) hostPath ignores fsGroup (kubernetes/kubernetes#138411), so /data/logs was created root-owned and non-root training/inference pods hit `PermissionError [Errno 13]` creating their per-run log dir (`os.makedirs('/data/logs/<run>')`). The #611 init-container chowned /data/shared but not /data/logs — the same class of bug on the logs volume. Extend the init (renamed init-shared-data -> init-writable-data) to chown+chmod BOTH hostPath volumes and mount both. Training and inference pods share one spec (job.yaml), so this covers both; ingestion already covered by the /data/shared chmod. Bump chart 1.9.16 -> 1.9.17. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): address review — sticky bit on shared dirs + reset-then-reuse parity Two reviewer follow-ups on #612: - chart: chmod the writable hostPath dirs 3777 (was 2777) — add the sticky bit so one writer can't unlink/rename another writer's files in /data/shared // /data/logs (/tmp semantics). setgid is retained. Safe given the uid topology (dir owned by 1000; training pods run as 1000; the ingestor writes its own subtrees as a stable uid) and no cross-uid filesystem deletes exist in client-runtime. Chart 1.9.17 -> 1.9.18. - install-k8s.ps1: the adopt/reconcile helm upgrade now prefers --reset-then-reuse-values when `helm upgrade --help` advertises it (Helm >= 3.14), falling back to --reuse-values otherwise — so NEW chart defaults reach adopted Windows edges on auto-upgrade (bash parity with install-client-helm.sh). manifest.sha256 regenerated. Tests: helm-unittest updated (3777); new Pester test asserts the reset-then-reuse preference; full Pester suite 452 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): add CAP_FSETID so the init's setgid bit survives (Bugbot) init-writable-data chowns the shared/logs hostPath dirs to GID 1000 then chmods 3777. With caps dropped to CHOWN+FOWNER only, the kernel silently strips S_ISGID on the chmod — after the chown the dir's group no longer matches the process (fsgid 0), and a root process without CAP_FSETID can't set setgid on it — so the mount landed at 1777 and new files did NOT inherit GID 1000 as documented. Add FSETID to the cap set; setgid now sticks. helm-unittest asserts FSETID present. Chart 1.9.18 -> 1.9.19. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): drop risky fsGroup + make init per-volume best-effort (Bugbot) Two Bugbot findings on the writable-volume fix: - HIGH — remove `fsGroup: 1000` / `fsGroupChangePolicy` from jobs-manager. It is a no-op on hostPath (kubelet ignores fsGroup — the init does the work) and on CSI it only grants jobs-manager's OWN processes GID 1000 while its OnRootMismatch relabel flips the shared/logs volumes to group 1000 — stripping the group-0 access the spawned training pods (UID 1001 / OpenShift arbitrary UID, GID 0) and the host-UID ingestion pods rely on (docs/SECURITY.md §5.3). It never reaches those spawned writers, so it was all regression risk and no gain. Those pods keep their own documented posture; CSI is untouched (matches develop). - MEDIUM — the init now fixes each dir INDEPENDENTLY and best-effort: `for d in /data/shared /data/logs; do chown && chmod || echo <warn>; done`. A chown that can't complete (e.g. /data/shared on an NFS root_squash export) no longer aborts the chain and skips /data/logs — the other dir is still repaired and jobs-manager still starts; a truly unwritable mount surfaces as a clear error at the writer pod instead of wedging the edge in Init. helm-unittest updated (no fsGroup on either path; per-dir loop; CSI skips init). Chart 1.9.19 -> 1.9.20. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What & why
Any user installing on a filtered corporate network can dead-end here — via the official one-liner, not just a dev build. A proxy or antivirus can truncate or block a GitHub-release binary mid-transfer;
install-k8s.ps1had a single transport (Invoke-WebRequest) and a single after-the-fact signal (the checksum), so a blocked k3d download stopped the install at the crypticSystem tool checksum verification failed. This is the field failure behind #578 ("installer must complete on restricted/corporate networks").This is not specific to how the installer is launched. The kubectl/k3d/helm downloads happen in Step 2 ("Install system tools") of
install-k8s.ps1, which runs identically no matter the entry point — the officialreleases/download/<tag>/install.ps1one-liner, a.tgz, or a dev branch. Step 2 always fetches the tools from the same upstream URLs (github.com/k3d-io/…,dl.k8s.io,get.helm.sh) over the user's own network, so the same proxy/AV that truncates the transfer fails the install regardless of source. (It happened to surface during testing, but the install method was not the cause — a real user on that network hits it from the official link.)Also: the k3d winget fallback was dead weight — k3d has no winget manifest, so
Rancher.k3dalways returned "No package found" before the direct download ran anyway.Closes#607. Contributes to #578.
PowerShell (
install-k8s.ps1)Get-VerifiedDownloadtries three transports in turn —Invoke-WebRequest→curl.exe→ BITS. A different HTTP stack commonly succeeds where one is blocked/truncated (the primary fix).Test-DownloadCompletevalidates each result before the checksum: present, ≥ a per-tool size floor, and the expected magic bytes (MZfor a.exe,PKfor the helm zip). A short / error-page / altered payload is caught as a transfer failure (distinct from a checksum mismatch) and the next transport is tried.Bash parity (
lib/common.sh,lib/setup-linux.sh)_assert_download_sizegates kubectl/k3d/helm on a size floor before the checksum, so a truncated/blocked transfer reports the real reason. (Linux already usescurl.)TB_MIN_DOWNLOAD_BYTESlets the bats fetch mocks — tiny fixture files — relax the floor; the real floor applies in production.Tests
Test-DownloadComplete(7 cases: complete .exe/.zip, truncated, too-small, wrong-magic, missing, magic-skipped) + a guard that the k3d winget branch is gone. Full Pester suite green (436)._assert_download_size(4 cases) + the env hook; existing fetch mocks updated to setTB_MIN_DOWNLOAD_BYTES=0.setup-linux/setup-macos/common/check-driftall green; assertion-enforce clean.For reference: the field symptom (official-link install on a corporate machine)
Not auto-fixed (by design)
If AV delivers a complete but rewritten binary (full size, valid magic, altered bytes), the checksum still fails — correctly — and the user gets the clear "proxy/AV may be rewriting the binary → add an exclusion" message. Antivirus-specific auto-exclusion and an internal tools-mirror override were deliberately left out of scope.
🤖 Generated with Claude Code
Note
Low Risk
Installer-only download/validation changes with parity tests; no runtime cluster or auth behavior changes.
Overview
Windows (
install-k8s.ps1) addsTest-DownloadComplete(size + optionalMZ/PKmagic) andGet-VerifiedDownload, which retries kubectl/k3d/helm fetches across Invoke-WebRequest → curl.exe → BITS, only accepting complete payloads before checksum. Failed transfers surface an allowlist/AV message instead of a generic checksum error. The dead k3d winget path is removed (no manifest).Linux/macOS adds
_assert_download_sizeincommon.shand calls it before checksum verification for kubectl, k3d, and Helm insetup-linux.sh, with the same transfer-vs-tampering distinction.TB_MIN_DOWNLOAD_BYTES=0in bats mocks keeps existing fetch tests working.Tests cover the new helpers;
manifest.sha256is updated for touched scripts.Reviewed by Cursor Bugbot for commit 1410766. Bugbot is set up for automated code reviews on this repo. Configure here.