Uh oh!
There was an error while loading. Please reload this page.
release-train: develop -> staging - #462
Merged
Merged
Conversation
…before Helm (#388) (#397) * feat(installer): Windows provisioning parity — login + client create before Helm (#388) Port the bash provisioning sequence (scripts/lib/provision.sh) to install-k8s.ps1, ending the legacy hand-copied Client-ID/password flow: - Step reorder: 3 = install the tracebloc CLI (was 5), 4 = register this machine, 5 = install the client via Helm. Roadmap updated. - Invoke-ProvisionClient (Step 4): browser sign-in (tracebloc login, device flow, attached to the console) + tracebloc client create --yes --credential-file — the credential never reaches the terminal; the file is parsed (KEY=value, first-'=' split) and deleted immediately. The minted slug becomes the namespace (bash Q2 parity). Includes the one-client-per-machine pre-flight (fail-closed on inconclusive reads; legacy 'tracebloc'-namespace deferral) and Print-CreateFailure (real failure reason incl. the carbon-zone special case). - Adopted re-runs reconcile in place: reuse the previous values password, heal a stale clientId to the adopted UUID; honest terminal error when the local values file is gone. - TRACEBLOC_CLIENT_ID/PASSWORD env pair stays as the unattended path (verified once, non-interactively). The old interactive prompts survive ONLY as the fallback for a missing/too-old CLI. - Get-InstalledClientInfo factors the one-client enumeration (#200 fail-closed semantics) so the pre-flight and the Helm guard share one source. CLI-install failure copy no longer claims the client is set up. Closes#388. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): adopted heal passes the one-client guard + paste-sanitize the name (Bugbot r1) - The one-client guard now lets ADOPTED mode through on an id mismatch — that mismatch IS the heal the mode exists for (helm still stores the cli#125-era numeric dashboard id while the backend anchored this cluster to the adopted UUID). Every other mode still refuses. The adopted Pester test now drives the realistic stale-id scenario, plus a negative test proving the guard is intact outside adopted mode. - ConvertTo-SanitizedInput strips CSI sequences, bracketed-paste markers, and control chars from the Step-4 name prompt (port of common.sh _strip_paste_garbage; customer-reported on the bash flow 2026-07-20). Pester: 118 passed / 0 failed / 8 skipped (Windows-only). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): verify minted creds, surgical adopted reconcile, signal-safe credential file (Bugbot r2) - Minted credentials now verify via api-token-auth like every other mode (never skip verification by provisioning method): backend skew or a mid-flow deactivated account fails at install, not as a crash-looping pod. Unreachable backend stays warn-and-continue. - Adopted mode with a LIVE release reconciles surgically: helm upgrade of THAT release, in ITS namespace, with --reuse-values --set-string clientId=<uuid> — preserving the deployed configuration and secret (bash parity) instead of regenerating values.yaml with fresh defaults. Needs no local password at all; the local values file gets only its clientId line healed. A rebuilt cluster (adopted anchor, no release) keeps the full values write and its honest no-password error. Wait-ForClientReady now watches the live release's namespace. - The credential file is removed in a try/finally spanning mint->parse, so Ctrl-C / terminating errors / Err exits can't leave the secret on disk (ps1 analogue of _PROVISION_CRED_FILE + install_cleanup). Pester: 119 passed / 0 failed / 8 skipped (Windows-only). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
… (backend#1252) (#400) * fix(installer): make the curl TLS floor structural, not per-call-site (backend#1252) `CURL_SECURE` was a bare constant every call site had to splice in by hand, so call sites kept losing it: seven live `curl` invocations ran with no minimum TLS version, including the POST in `verify_credentials()` that carries the client's password. These installs run on customer-managed hosts and behind TLS-inspecting proxies, which negotiate down to whatever the client permits — the reason this repo adopted an explicit floor instead of trusting curl's defaults. Add `curl_secure()` in `scripts/lib/common.sh` and route every fetch in `scripts/lib/*.sh` through it (18 call sites). The wrapper always passes `--tlsv1.2` and supplies default `--connect-timeout 30` / `--max-time 300`. Defaults are injected before `"$@"`, so a call site that wants a tighter bound still wins (curl honours the last occurrence), and a transfer that bounds itself with `--speed-limit`/`--speed-time` gets no injected `--max-time` — a hard deadline would fail a slow-but-healthy link on a large binary download. Every existing site therefore keeps its effective behaviour; seven gain the floor and nine previously unbounded ones gain a deadline. Also fixed while here: - `gpu-amd.sh` had the least-bounded curl usage in the repo — no TLS floor, no timeout, no retry. Both calls now go through the wrapper; the `.deb` download is retry-wrapped. The listing scrape deliberately is not: `retry()` reports attempts on stdout, which is that function's return value. - `install-k8s.ps1`'s WSL2 here-string had the same two nvidia-container-toolkit fetches bare. It cannot source `common.sh`, so it spells the flags out inline the way the bootstrap does. `scripts/install.sh` keeps its seven hardcoded literals: it is the trust root that fetches `common.sh`, so it cannot source the wrapper. `CURL_SECURE` stays defined and unchanged for out-of-tree callers, but nothing in the repo reads it now — the wrapper names the flag itself, so the constant can never silently reshape every fetch in the installer. Enforcement: an INTERIM third check in `scripts/check-style.sh` fails on a bare `curl`. tracebloc/.github#65 already implements this properly (a shell-aware lexer, not a grep) in a shared reusable workflow, but that workflow is not on `main` yet and cannot be referenced from here until it is. The check is marked for retirement the moment this repo adds that caller. Regenerated `scripts/manifest.sha256` (R8 supply-chain gate). Found in #399. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): stall-bound the kubectl fetch, don't give it a deadline (Bugbot) `_fetch_kubectl` had no time bound at all, so routing it through `curl_secure` handed it the wrapper's default `--max-time 300`. kubectl is a ~50 MB binary, and this repo already documents (at `_fetch_k3d_release`, same file) that a fixed ceiling fails a slow-but-healthy link at that size — so the wrapper would have made every retry fail where the fetch previously completed. Give both fetches the same `--connect-timeout 15 --speed-limit 1024 --speed-time 60` as the k3d pair. That is also how `curl_secure` knows to skip its default deadline, and it is strictly better than before: the fetch was previously unbounded in both directions, so a mid-stream stall hung the step indefinitely. Audited the other 7 sites that now inherit the 300s default — get.docker.com, get-helm-3, the Homebrew script, stable.txt, the DMG checksum, the device-plugin manifest and the amdgpu-install package are all small text/script payloads. The only large downloads in the repo are kubectl, k3d and the Docker Desktop DMG; the latter two were already stall-bounded. Adds a bats test pinning it, since nothing covered `_fetch_kubectl` before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
…hes edges (#398) The prod ingestor digest pin reached no customer edge, for two independent reasons. 1. Standard installs never applied the overlay. `client/values-prod.yaml` was an install-time `-f` overlay, but the installer runs `helm upgrade --install … --values "$values_file"` with only its generated values file. No installer path ever passed the overlay, so a normal prod install kept the base default `images.ingestor.digest: ""` and floated on tag `0.7` with imagePullPolicy=Always — exactly like dev and staging. 2. Where it was applied by hand it could never be updated. The fleet auto-upgrade CronJob runs `helm upgrade --reset-then-reuse-values` with no `-f` and no `--set`: that resets to the new chart's `values.yaml` defaults, then re-applies the release's stored user-supplied values. An overlay value is user-supplied, so it was replayed verbatim forever — and Helm only auto-reads `values.yaml` from a chart, so the updated overlay shipped inside the new chart archive was never read. Chart defaults propagate through that upgrade; user-supplied values freeze. So the pin has to be a chart default, which is how the egress-proxy squid image has always been pinned. * `images.ingestor.prodDigest` (new, chart default) carries the pin. * `images.ingestor.prodPin` (new, default true) gates it. A new `tracebloc.ingestorDigest` helper resolves one effective digest for every consumer: explicit `images.ingestor.digest` wins in any environment, else `prodDigest` when prodPin is on and the resolved CLIENT_ENV is prod, else empty (float on `tag`). dev/staging carry `env.CLIENT_ENV: dev|stg` in user-supplied values while the template defaults it to prod, so prod pins and non-prod floats with zero per-edge action. `prodPin: false` floats a canary prod edge. * The metadata-backfill hook now shares the same helper, so it can never run a different build than the ingestions it reconciles. * The consuming contract in client-runtime `_build_image_reference` is unchanged: digest set -> repo@digest + IfNotPresent, empty -> repo:tag + Always. `client/values-prod.yaml` is removed rather than kept as a shim: passing `-f values-prod.yaml` is what created the frozen-value bug, so leaving the file would invite operators to re-create it. `MIGRATION.md` documents the move, the canary knob, and how to clear a hand-layered stored digest (which still takes precedence and would otherwise stay frozen). `resolve-ingestor-digest.sh` keeps its resolution mechanism (multi-arch index digest + guard) and now writes `prodDigest` in `client/values.yaml`, keyed on that unique name with a single-match assertion so it can never rewrite a sibling image's `digest:` leaf, and reminding the operator to bump Chart.yaml. CI's `ingestor-multiarch` guard now validates `prodDigest` too — it previously read only values.yaml and so never checked the overlay it was meant to guard — and hard-fails an empty pin, which would silently un-pin prod. Verified on a throwaway k3d cluster: installing the published 1.9.5 chart renders an empty digest (prod floats today), and `--reset-then-reuse-values` to this tree lands the chart-default pin on the already-installed edge, while the same upgrade leaves an overlay-style user-supplied pin frozen on its install-day value. `scripts/tests/e2e-auto-upgrade.sh` now asserts all of that plus the prodPin opt-out surviving the next auto-upgrade. Chart bumped 1.9.5 -> 1.9.6 — the chart only publishes on a version change, so an unbumped pin refresh reaches no edge. Fixestracebloc/backend#1245 Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…395) The Tier 0 fast path skips install_system_deps but still runs install_helm, which shells to get-helm-3 — that needs openssl (checksum) and tar (unpack). On a minimal docker-group host lacking them, the "zero-privilege" install promised no admin, then failed mid-Helm-install. Add _ensure_helm_prereqs: verify openssl+tar are present before install_helm on the Tier 0 path. We cannot install them there without sudo (Tier 0 promises no admin, and setup_pm/$PM_INSTALL aren't even run), so surface the real constraint with an actionable fail-fast instead of silently sudo-ing or dying cryptically. The full flow is unchanged (install_system_deps still installs them). Regenerate scripts/manifest.sha256 for the setup-linux.sh change. Co-authored-by: shujaat hasan <shujaathasan@shujaats-MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Lukas Wuttke <lukas@tracebloc.io>
…load, no get-helm-3 (#396) * feat(installer): self-serve Helm prerequisites — verified direct download, no get-helm-3 (#395) The installer now takes care of its own requirements instead of erroring out and telling the user to install tools: - install_helm fetches the pinned Helm release directly from get.helm.sh and verifies it against the published .sha256sum (fail-closed), exactly like the k3d direct download (#382). helm's get-helm-3 script is gone — it floats on the mutable helm/helm@main, performs its checksum step with openssl (absent on minimal cloud images, Bugbot #383), and its fetches are unbounded. openssl is no longer needed anywhere in the flow. - HELM_VERSION pin in common.sh (v4.2.3; 'latest' resolves at install time via get.helm.sh/helm-latest-version, same verified path; malformed tags fail closed before any fetch). - _ensure_unpack_tools: when tar/gzip are genuinely missing on the Tier 0 fast path (which skips install_system_deps), install them via the package manager — quietly as root/passwordless sudo, with an honest one-line reason when a password is needed — rather than aborting with 'go install tar'. install_system_deps drops openssl, adds gzip. Supersedes the fatal-error approach in #395 (product call: never tell the user to install tools we can install ourselves). Windows already fetches Helm directly from get.helm.sh, so the ps1 path is untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): route sudo probes through _real_sudo/_have_sudo_bin (Bugbot r1) The A2 sudo shadow runs '-n true' as a command when root, and 'has sudo' is always true because the shadow function exists — use the #372 primitives like preflight_sudo/_probe_privilege do. The root-without- sudo-binary strip is dropped entirely: the shadow already handles root by executing the command directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): Tier 0 unpack install — sudo keepalive + dpkg-lock wait + one combined install (Bugbot r2) _ensure_unpack_tools ran package installs on the Tier 0 path without the full flow's guards: apt could sit on the dpkg lock invisibly behind the spinner, and a long wait could outlast the just-primed sudo timestamp so the next sudo re-prompts behind the spinner and hangs. Prime, then keep the ticket warm (preflight_sudo's pattern; killed right after the install — the zero-privilege tier shouldn't hold a warm admin ticket), wait out the dpkg lock (bounded + visible), and install everything in ONE package- manager call so there's a single sudo consumer right after priming. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): keepalive ownership + latest-tag isolation (Bugbot r3) - _ensure_unpack_tools kills only the keepalive IT started: on the Tier 1/2 recovery path SUDO_KEEPALIVE_PID belongs to preflight_sudo, and killing it would let later privileged steps re-prompt behind a spinner. The global is only claimed when empty (install_cleanup coverage) and only cleared when it is ours. - HELM_VERSION=latest: isolate the endpoint body with tail -1 — retry's attempt notices go to stdout and a failed-then-successful fetch would concatenate them into the captured tag, failing the anchored regex with a false 'couldn't resolve'. (install_k3d's resolver is immune: its ${var##*/} strip discards anything before the redirect URL.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(copy-catalog): regenerate 00-install golden for the HELM_VERSION knob The PR adds a HELM_VERSION env var (common.sh), which surfaces one line in install-k8s.sh --help. emit_install reads --help live, so the golden drifted by exactly that line. Regenerated via TB_UPDATE_GOLDEN=1; verified locally (bats copy-catalog now green). * fix(installer): drop obsolete _ensure_helm_prereqs — Tier 0 no longer blocks on openssl (Bugbot #396) get-helm-3 is gone (replaced by a sha256-verified direct download), so Helm no longer needs openssl. The Tier-0 preflight still demanded openssl+tar and failed fast, aborting minimal-image installs for a dependency Helm doesn't use — and pre-empting _ensure_unpack_tools, which installs tar/gzip Tier-0-aware. Remove the function and its Tier-0 call; drop the tests that encoded the old openssl preflight (tar/gzip stays covered by _ensure_unpack_tools' own tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
… client) (#381) * feat(install): prepare-host — one-time admin step so researchers install unprivileged (#1178) Adds the admin-only prepare-host path: installs the container runtime + prerequisites and grants a named non-admin user docker-group access, so they later install tracebloc at Tier 0 with no root. Rebased onto the current feat/lpi-tier0 (which now targets develop after routing #374 merged); collapsed from the prior 4-commit history to avoid replaying an intermediate that carried resolved conflict markers. Incorporates the three Bugbot fixes from that history: - reap the sudo keepalive on EXIT (no lingering background sudo -v loop); - grant docker-group ONLY to the explicit TB_PREPARE_USER, never $SUDO_USER (the admin who ran prepare-host); - only print the "researcher can now install with no admin" message when a docker-group grant actually succeeded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * prepare-host: sudo-verified engine gate; never sg-re-exec or grant the admin (Bugbot) install_docker_engine's tail assumed the invoking user is the end-user: on the documented non-root prepare-host path it added the ADMIN to the docker group and the sg re-exec re-ran the script WITHOUT the prepare-host argument -- a silent FULL provision as the admin -- and a socket-less admin aborted before the TB_PREPARE_USER grant. In TB_PREPARE_HOST_MODE the daemon check runs via sudo, the group-add and re-exec are skipped, and only the researcher named by TB_PREPARE_USER is ever granted the socket (same least-privilege rule as #377). Two bats regressions pin both escapes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(prepare-host): reach the sub-script on healthy machines + terminal active-daemon corner (Bugbot r2) Two follow-ups on the prepare-host re-land: - install.sh: prepare-host/--prepare-host now skip the healthy-install bailout — the argument is useful precisely on a machine that is already set up (grant ANOTHER researcher docker-group access), and the bailout would open the home screen and silently skip the TB_PREPARE_USER grant. It skips ONLY the bailout: the force flag is split out (_tb_force), so host-prep is never mislabeled as a forced reinstall — a stale sub-script without the prepare-host dispatch would otherwise treat the run as a full forced provision. - setup-linux.sh: in prepare-host mode, a daemon that systemd reports ACTIVE but that doesn't answer 'sudo docker info' is now terminal with prepare-host-appropriate guidance, instead of falling through to the docker-group 'log out and back in' abort meant for regular users. Regression tests: bootstrap bats stamps DEFAULT_REF like the release pipeline (assignment line only) to prove prepare-host passes the bailout with TB_FORCE_REINSTALL unset; setup-linux bats covers the active-but- unresponsive daemon corner. 16/16 + 58/58. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prepare-host): terminal daemon-down path + trim TB_PREPARE_USER (Bugbot r3) - Daemon DOWN in prepare-host mode: start it via sudo (starting Docker IS host preparation), re-verify, and stay terminal in prepare-host wording on every failure — the shared diagnostics end with 're-run this installer', which for the admin means a full provision as themselves, the exact outcome prepare-host exists to prevent. Reboot-required case gets prepare-host wording too. - TB_PREPARE_USER is trimmed before the non-empty gate and the grant, so a pasted value with stray spaces doesn't fail usermod while skipping the honest no-grant messaging. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prepare-host): dispatch at any argument position (Bugbot r4) install.sh's bailout exemption scans all of "$@" for prepare-host, but the sub-script only dispatched on $1 — so '--force prepare-host' skipped the bailout, exported the force flag, and then fell through into a FULL FORCED provision as the admin. The dispatch now scans every argument, matching the bootstrap's contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prepare-host): enable docker on boot in the already-running path (Bugbot r5) Running NOW isn't host-prep: after a reboot the Tier-0 researcher can't start the daemon themselves. The success path now ensures docker.service is enabled on boot (best-effort), matching the fresh-install and daemon-down recovery paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prepare-host): don't let diagnostics pipelines abort before the guidance (Bugbot r6) systemctl status exits 3 for an inactive unit (and the shared block's grep exits 1 on no match) — under set -e -o pipefail the failing diagnostics pipeline aborted BEFORE the terminal error message, a silent death with no re-run guidance. Both the prepare-host block and the shared daemon-down block now guard the pipeline with || true; the bats stub returns the real rc 3 to exercise the path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ler (#401) * fix(installer): make the curl TLS floor structural, not per-call-site (backend#1252) `CURL_SECURE` was a bare constant every call site had to splice in by hand, so call sites kept losing it: seven live `curl` invocations ran with no minimum TLS version, including the POST in `verify_credentials()` that carries the client's password. These installs run on customer-managed hosts and behind TLS-inspecting proxies, which negotiate down to whatever the client permits — the reason this repo adopted an explicit floor instead of trusting curl's defaults. Add `curl_secure()` in `scripts/lib/common.sh` and route every fetch in `scripts/lib/*.sh` through it (18 call sites). The wrapper always passes `--tlsv1.2` and supplies default `--connect-timeout 30` / `--max-time 300`. Defaults are injected before `"$@"`, so a call site that wants a tighter bound still wins (curl honours the last occurrence), and a transfer that bounds itself with `--speed-limit`/`--speed-time` gets no injected `--max-time` — a hard deadline would fail a slow-but-healthy link on a large binary download. Every existing site therefore keeps its effective behaviour; seven gain the floor and nine previously unbounded ones gain a deadline. Also fixed while here: - `gpu-amd.sh` had the least-bounded curl usage in the repo — no TLS floor, no timeout, no retry. Both calls now go through the wrapper; the `.deb` download is retry-wrapped. The listing scrape deliberately is not: `retry()` reports attempts on stdout, which is that function's return value. - `install-k8s.ps1`'s WSL2 here-string had the same two nvidia-container-toolkit fetches bare. It cannot source `common.sh`, so it spells the flags out inline the way the bootstrap does. `scripts/install.sh` keeps its seven hardcoded literals: it is the trust root that fetches `common.sh`, so it cannot source the wrapper. `CURL_SECURE` stays defined and unchanged for out-of-tree callers, but nothing in the repo reads it now — the wrapper names the flag itself, so the constant can never silently reshape every fetch in the installer. Enforcement: an INTERIM third check in `scripts/check-style.sh` fails on a bare `curl`. tracebloc/.github#65 already implements this properly (a shell-aware lexer, not a grep) in a shared reusable workflow, but that workflow is not on `main` yet and cannot be referenced from here until it is. The check is marked for retirement the moment this repo adds that caller. Regenerated `scripts/manifest.sha256` (R8 supply-chain gate). Found in #399. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): stall-bound the kubectl fetch, don't give it a deadline (Bugbot) `_fetch_kubectl` had no time bound at all, so routing it through `curl_secure` handed it the wrapper's default `--max-time 300`. kubectl is a ~50 MB binary, and this repo already documents (at `_fetch_k3d_release`, same file) that a fixed ceiling fails a slow-but-healthy link at that size — so the wrapper would have made every retry fail where the fetch previously completed. Give both fetches the same `--connect-timeout 15 --speed-limit 1024 --speed-time 60` as the k3d pair. That is also how `curl_secure` knows to skip its default deadline, and it is strictly better than before: the fetch was previously unbounded in both directions, so a mid-stream stall hung the step indefinitely. Audited the other 7 sites that now inherit the 300s default — get.docker.com, get-helm-3, the Homebrew script, stable.txt, the DMG checksum, the device-plugin manifest and the amdgpu-install package are all small text/script payloads. The only large downloads in the repo are kubectl, k3d and the Docker Desktop DMG; the latter two were already stall-bounded. Adds a bats test pinning it, since nothing covered `_fetch_kubectl` before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): stop the ROCm package lookup from aborting the installer `_find_package_name` ran `curl … | grep … | head -1` as a single pipeline and returned its status. `install-k8s.sh` sources this lib under `set -euo pipefail`, so that pipeline could kill the installer two different ways: 1. A failed fetch (404, timeout, proxy block) made the command substitution non-zero, the caller's assignment inherited it, and `set -e` aborted BEFORE the friendly `[[ -z "$name" ]] && error "No amdgpu-install …"` on the next line could run. The user got a silent abort mid-GPU-step instead of an actionable message, and the RHEL major-version fallback was unreachable for the same reason. 2. `head -1` can close the pipe while grep is still writing, so grep takes SIGPIPE (141) and `pipefail` propagates that as a pipeline failure even though a filename WAS found. It only triggers when the directory index exceeds the pipe buffer, so it fails on large mirrors only. Capture the fetch and the match separately and let neither fail the function, then take the first match with `${var%%…}` so `head` leaves the pipeline entirely. The contract is unchanged — filename on stdout, nothing when not found — so emptiness remains the single signal all three callers already test. Adds scripts/tests/gpu-amd.bats (first coverage for this lib): the contract, both hazards, and a caller-shaped regression test under `set -euo pipefail`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ci: re-trigger standard-checks after base retarget to develop (#401) Retargeting the PR base from #400's merged branch to develop doesn't fire a pull_request event, so standard-checks (Unit tests + Lint) never ran on this head. Empty commit fires synchronize so the required checks run against develop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
Cursor Bugbot reviews this repo with zero project context today — no repo in the org has a BUGBOT.md. On `client` Bugbot writes ~21.6x the human inline review volume, so it is effectively the whole code review here; every house rule it has to re-derive is a pass it doesn't spend on a hard finding. Encodes the invariants that recurring findings actually cluster on, each with the reason and a real reference in this repo: the `$CURL_SECURE` TLS floor being a constant rather than a wrapper (so new curl calls lose it silently), kubectl/helm/curl timeout conventions, version-before-URL validation, fail-closed guards (`detect_installed_client`), never reporting success you haven't verified (SEAL-CHECK), Helm nil-guards on new values keys across both `--reuse-values` and `--reset-then-reuse-values`, the `resource-policy: keep` stored-manifest trap that cost a PVC set, digest pinning incl. the prod-overlay CI blind spot, and the R8 manifest regeneration step. Also records verified non-issues so Bugbot stops re-reporting them — most importantly that `scripts/lib/*.sh` omit `set -euo pipefail` by design (sourced after install-k8s.sh sets it), the deliberate `set -uo pipefail` guard scripts, and the known cross-file SC2034 false positive. Item 4 of tracebloc/backend#930; improves on RFC 0001's Appendix A draft. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…oc/backend#930) (#402) GitHub auto-closes an issue in another repository only when the PR body names it owner-qualified. A bare `repo#N` merely cross-references and closes nothing -- and the template's own hint taught `Ref tracebloc/other-repo#456`, which is not a closing keyword at all. Eight code-complete issues stayed open for days-to-weeks this way (tracebloc/backend#1171-#1176, #376, tracebloc/cli#393), dragging two epics to 0% and 14% when the true figures were 67% and 24%. Someone had to notice and close all eight by hand. Makes the qualified form the visible example in Related, and adds one checklist item. No other changes -- a template nobody reads because it grew is worse than the bug. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…back-link (#403) RFC numbers are assigned per repo, so a bare "RFC 0001" names four different documents across the org. Add the qualified ID (`RFC-CLIENT-0001`) to the header, in the bold-label style this document already uses, and give it an explicit `Draft` status. Also repoint the `**RFC:**` back-link. It pointed at `./0001-least-privilege-install.md`, which (a) does not exist on develop and (b) collided with this very file's number. That document is being renumbered to `0002-least-privilege-install.md` in #369, so the link now targets 0002. The link is dead either way until #369 merges — it is dead on develop today. If this lands first it stays dead for the gap; if #369 lands first it resolves immediately. No ordering requirement. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…lly (#404) The post-upgrade Helm hook (#380) coupled a one-time, best-effort data migration to every chart upgrade, which repeatedly surfaced timing and --timeout edge cases. The backfill is a one-time-per-cluster operation, so run it manually as a standalone Job instead. The tracebloc-backfill runner (data-ingestors #393/#395) is unchanged and remains the mechanism. Removes: - templates/metadata-backfill-hook.yaml - tests/metadata_backfill_hook_test.yaml - metadataBackfill block in values.yaml + values.schema.json Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…e-local reuse) (#405) * fix: Bugbot findings from promotion PR #383 (helm probe timeout + node-local reuse) A) Ungated Helm in client probe: Get-InstalledClientInfo ran helm list/get with no bounded kubectl probe, so a wedged API server hung Step 4/5 (helm has no request timeout). Add Test-ApiReachable (5s-bounded kubectl probe, mirrors Get-TrainingResources) and gate the enumeration behind it — an unreachable API now degrades to ListUnknown instead of hanging. B) Reuse ignores node-local data: under TB_STORAGE_MODE=node-local there is no /tracebloc host bind-mount, so "reuse = keep and adopt" is false. Make the leftover-guard prompt option and the reuse branch honest under node-local (data left on disk, NOT adopted; cluster starts empty in-node). Hostpath behavior unchanged. Regenerate scripts/manifest.sha256 for the cluster.sh + install-k8s.ps1 changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: node-local leftover-guard follow-ups from Bugbot on #405 A) Keep reply rejected under node-local: the node-local prompt shows "[r] keep", but the parser only accepted r/reuse, so typing "keep" aborted the install. Lowercase the reply (tr, bash 3.2-safe) and accept r/reuse/k/keep -> reuse. B) Non-interactive reuse still claimed adopt: the no-TTY recovery guidance advertised --reuse-data as "adopt the existing data" for every mode. Make it storage-mode-aware — under node-local it keeps data on disk, NOT adopted (cluster starts empty in-node), matching the interactive reuse branch. Regenerate scripts/manifest.sha256 for the cluster.sh change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: shujaat hasan <shujaathasan@shujaats-MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…version + Windows leftover-data guard) (#406) * fix: Bugbot findings from promotion PR #383 (probe timeout + kubectl version + Windows leftover-data guard) A) Probe hangs on wedged Docker (High): _probe_runtime_usable ran bare `docker info` (no timeout) on every install, so a wedged daemon hung a headless SSH install forever. Bound it with timeout/gtimeout when present (fallback to bare call); read-only and never fatal. B) Retry notices corrupt kubectl version (High): KUBE_VER captured retry's stdout notices, polluting the version and breaking the download URL. Isolate it the way the Helm resolver does — tail -1 + tr + a version-tag regex that fails closed on an unresolvable value. C) Windows missing leftover-data guard (Medium): New-K3dCluster silently adopted prior data. Port guard_leftover_data to PowerShell (detect flat + per-release layouts, prompt reuse/wipe/new/abort, honor TB_LEFTOVER_ACTION / HOST_DATA_DIR / TRACEBLOC_SKIP_LEFTOVER_GUARD, non-interactive fail-safe abort). Windows is hostpath-only (node-local has no Windows path). Refactored HOST_DATA_DIR validation into a shared Confirm-DataDir. Regenerate scripts/manifest.sha256 (probe.sh, setup-linux.sh, install-k8s.ps1). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: cross-platform Pester paths + nested-junction-safe wipe (#406 follow-ups) 1) Pester (ubuntu-latest) failed: the new leftover-data-guard tests hardcoded Windows '\' separators, which are literal chars (not separators) under Linux pwsh, so Get-LeftoverDataDirs (Join-Path -> '/') never matched the expected paths. Build all test paths with Join-Path / [IO.Path]::Combine so they pass on both Windows and Linux pwsh. Behavior/coverage unchanged. 2) Wipe could follow nested junctions (Bugbot r3655703571): Remove-LeftoverData guarded only the TOP-LEVEL reparse point, then `Remove-Item -Recurse`, which on Windows PowerShell 5.1 descends into nested junctions and can delete targets OUTSIDE HOST_DATA_DIR. Add Remove-TreeNoFollow (rm -rf semantics): walk depth-first and unlink any reparse point without descending, so a nested junction is removed but its target is never touched. Add a Pester test for it. Regenerate scripts/manifest.sha256 for the install-k8s.ps1 change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: refresh stale BUGBOT.md guidance (curl_secure + prodDigest) — Bugbot #383 A) Curl TLS section described CURL_SECURE as a hand-spliced constant with a stale "already missing" list. The real rule is the curl_secure() wrapper in common.sh, enforced by check-style.sh rule 3 ("no bare curl"). Rewrote the bullet to flag bare curl bypassing curl_secure(), with the real exemptions. B) Prod overlay section told reviewers CI ignores client/values-prod.yaml and to flag overlay digest edits. That overlay was deleted; the fleet-wide prod pin moved to the chart default images.ingestor.prodDigest, which ingestor-multiarch (helm-ci.yaml) reads and hard-fails on (empty or single-arch). Rewrote to the real contract. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: shujaat hasan <shujaathasan@shujaats-MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…roaming-profile machines (#409) (#437) PowerShell background jobs spawn their runspace in $HOME; on managed machines with roaming profiles that is a UNC share, and every cmd.exe a job starts there prints 'CMD.EXE was started with the above path as the current directory. UNC paths are not supported.' plus a RemoteException error record — 2-6 alarming red blocks on an otherwise healthy install. All five Start-Job sites (wsl --update, wsl --set-default-version, wsl --list, NCT install, NCT verify) now pass a shared -InitializationScript that pins the job to $env:SystemRoot (always local; no-op off-Windows). An AST-based Pester gate fails any future Start-Job that forgets it. Closes#409 Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…lookups (re-land #410 onto develop) (#446) * fix(installer): pin k3d + helm versions — drop api.github.com latest lookups (#410) Ports the bash pins (#382) to the Windows installer: K3D_VERSION defaults to v5.9.0 and HELM_VERSION to v4.2.3 (lockstep with scripts/lib/common.sh until #435 single-sources them), env-overridable, validated against a release-tag shape before any URL is built (path-traversal gate mirroring the bootstrap and cli install.ps1). The unauthenticated releases/latest API allows 60 req/hour per IP — one shared corporate NAT exhausts it and fails installs (observed live on a customer install 2026-07-27). The literal value 'latest' still works but resolves API-free: the /releases/latest redirect Location for k3d, and get.helm.sh/helm-latest-version for helm — exactly like lib/setup-linux.sh. A Pester gate now fails the suite if any https://api.github.com fetch reappears in the installer. Closes#410 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): restore retry parity on the 'latest' version resolvers (Bugbot #438) The new API-free 'latest' resolvers ran a single request — the old lookups and lib/setup-linux.sh retry 3x5s. Resolve-ToolVersion now drives the resolver through Invoke-WithRetry (resolvers throw on failure), so a one-off network blip on flaky corporate egress retries instead of aborting the install; a persistent failure still fails closed with the pin-a-tag remedy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): bound the 'latest' lookup requests (-TimeoutSec 30, Bugbot #438) A host that accepts the TCP connect but never responds would hang the version resolvers indefinitely; the bash peers bound this with --connect-timeout 15 --max-time 30. Both resolver requests now carry -TimeoutSec 30; a timeout throws, so the retry ladder + fail-closed remedy from the previous commit take over. AST-based Pester gate keeps the timeouts in place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Lukas Wuttke <lukas@tracebloc.io> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…447) * fix(installer): fail fast on k3d spawn failure; bound the create wait (#439) * fix(installer): fail fast on k3d spawn failure; bound the create wait (#412, #426 Windows half) Two fixes to New-K3dCluster's create path, both observed live: 1. Start-Process failure ('%1 is not a valid Win32 application' from a broken k3d.exe) is non-terminating by default, leaving $k3dProc null — and 'while (-not $null.HasExited)' is always true, so the installer spun 'Creating compute environment...' forever over a dead install. The spawn now runs under -ErrorAction Stop in a try/catch that cleans up the temp logs + proxy config and fails with the real exception, the log path, and a remedy. 2. k3d cluster create --wait has no timeout of its own, so a stalled image pull spun the spinner indefinitely. The wait is now bounded (15 min default, TB_CREATE_TIMEOUT_MIN override): on expiry the process is killed, the last stderr lines and the install-log path are printed, and the install fails loudly. Extracted as Wait-ProcessWithDeadline so the deadline/kill path is unit-tested. Closes#412. Windows half of #426 (the bash half — k3d create + helm timeouts in cluster.sh / install-client-helm.sh — follows separately). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): remove the partial cluster when the create wait times out (Bugbot #439) Killing k3d mid --wait skips its own rollback, so the timeout path left a half-created cluster behind — and the next run's reuse path would see serversRunning > 0 and print 'Compute environment already running' over a broken environment. The timeout path now deletes the partial cluster (bounded at 2 min via Wait-ProcessWithDeadline) before failing, and tells the operator the exact manual command if the delete itself fails. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * fix(installer): Docker engine wait — 10 minutes, elapsed progress, named failure state (#440) * fix(installer): Docker engine wait — 10 minutes, elapsed progress, named failure state (#413) A first-ever Docker Desktop start on AV-heavy corporate machines routinely needs 5-10 minutes (WSL bootstrap, image unpack). The old 3-minute cap turned a normal cold start into a failed install plus a manual re-paste of the one-liner — observed as a recurring wait-then- re-run loop on hospital installs. - Wait bound: 3 min -> 10 min default, TB_DOCKER_WAIT_MIN override. - After the first minute the spinner shows elapsed minutes and the expected worst case, so the wait doesn't read as a hang. - On expiry the failure names the observed state: Docker Desktop process gone (start it / fix its error window) vs. running with the engine still down (tray-icon guidance) - instead of one generic line. Closes#413 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): exited-Docker path drops the slow-start advice (Bugbot #440) When Docker Desktop's process has exited, 'a first start can be slow' and the TB_DOCKER_WAIT_MIN hint contradict the diagnosis and steer operators toward raising the wait instead of restarting/fixing the crash. The slow-start reassurance + override hint now print only on the engine-still-starting path; the exited path fails with its own start-and-fix remedy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
#441) * fix(installer): run the network-FS guard before the log dir is created (#432) setup_log_file mkdirs HOST_DATA_DIR and tees the whole session's output onto it BEFORE run_preflight fires — so on the exact machine the network-FS guard was built for (NFS home + sudo + root_squash), the unguarded mkdir failed with a bare error, or the log dir landed squashed/nobody-owned, before the friendly named failure could print. New early_data_dir_guard runs right after validate_config and before setup_log_file: same filesystem classification (extracted as the shared _pf_is_network_fstype), console-only, silent on local/undetermined filesystems, defers to the full check's warning under TRACEBLOC_ALLOW_NETWORK_FS. The call is declare -F-guarded so a stale bootstrap without the new helper proceeds as before. Closes#432 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): early NFS guard names a followable remediation (Bugbot #441) The guard advised HOST_DATA_DIR=/local/path, but validate_config requires the data dir under $HOME (Bugbot #384 hardening) — so the exact audience this guard exists for (network home) was pointed at a fix that fails validation on re-run. The copy now states the real constraint and the two workable paths (local-home user, or the explicit TRACEBLOC_ALLOW_NETWORK_FS=1 override with its risk), and notes that datasets may stay on network storage via HOST_DATASET_DIR. Test asserts the impossible advice stays gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): early NFS guard skips an existing data dir (Bugbot #441 r2) The early guard's job is protecting the pre-log mkdir; an existing data dir has no at-risk mkdir, and a healthy machine's re-run must keep reaching the assess hand-off exactly as it did when the network-FS check lived only in run_preflight. Existing dir -> silent pass; the full preflight guard still classifies network storage for real (re)installs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…angs (bash half of #426) (#442) * fix(installer): bound k3d create and the helm calls — no indefinite hangs (#426, bash half) k3d cluster create --wait had no deadline of its own and its spinner none either: a stalled image pull (rate-limited registry, TLS-intercepting proxy) span the create forever. Both helm invocations (install + reconcile) ran under the deadline-less spin_cmd and could hang the same way against a wedged kube-apiserver. - create: --wait now always pairs with --timeout (default 15m, TB_CREATE_TIMEOUT_MIN override — the same env knob the Windows installer adopted); k3d aborts with a real error that the existing failure path dumps. spin() gained an optional deadline, used as a +5min backstop in case k3d itself wedges past its own timeout. - helm install + reconcile: new spin_cmd_bounded (spin_cmd + hard deadline, rc 124 with an explicit timeout note + log tail), default 10m via TB_HELM_TIMEOUT_MIN. Completes #426 (the Windows half shipped with the #439 stack). Closes#426 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): spin's deadline kills the wrapper's children too (Bugbot #442) The create backstop passes the `( k3d … )` wrapper-subshell PID into spin, and the deadline path signalled only that PID — orphaning k3d, which kept creating the cluster after the installer had already exited as failed, racing any retry. The deadline path now TERMs the children while the parent is still alive (afterwards they reparent to init and pkill -P can't see them), then the wrapper, with a KILL sweep after the grace period. Harmless when bash exec-optimizes the wrapper away. Test proves the child dies with the wrapper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): reparent-safe KILL sweep + explicit create-timeout UX (Bugbot #442 r2) 1. spin's deadline path captured no child PIDs, so once the TERM'd wrapper died, its children reparented to init and the pkill -KILL -P sweep found nothing — a TERM-immune k3d could survive the backstop. Child PIDs are now captured BEFORE any signal and the KILL sweep addresses them directly; test proves a trap-''-TERM child dies. 2. When the create backstop fires, the k3d log is often empty (hung daemon) — the operator saw a bare failure with no timeout hint. The 124 path now names the timeout, points at TB_CREATE_TIMEOUT_MIN, and deletes the partially created cluster (bounded) so a re-run cannot adopt it via the 'already exists' branch — parity with the Windows fix on #439. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): spin's deadline path is set -e-proof (Bugbot #442 r3) Bare pkill (returns 1 with no children), kill on a reaped pid, and wait after a kill (reports the signal) could each abort the deadline path before 'return 124' under the installer's set -e — callers would see 143/1: no timeout copy, no partial-cluster cleanup, no bounded note. Every signal step is now failure-proofed, and a bash -c 'set -euo pipefail' end-to-end test pins the 124 contract for the childless (exec-optimized) case that reproduces the abort. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): helm timeout names the pending-release unwedge (Bugbot #442 r4) A SIGKILLed helm (spin_cmd_bounded deadline) can leave the release wedged as pending-install/pending-upgrade — the next run then fails with Helm's 'another operation is in progress' and no guidance. Both call sites now capture the rc (if-! discarded it) and, on 124, print the exact unwedge commands (uninstall for a half-installed release, rollback for an upgrade) before the error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(installer): fix over-broad grep assertions on the helm-bound tests Count invocation/hint lines, not comment mentions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): rollback hint names the release, not the namespace (Bugbot #442 r5) The adopt path tracks release ($_rel) and namespace ($_ns) separately; the reconcile-timeout unwedge hint said 'helm rollback $_ns', which is a non-working command whenever they differ. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): base-10-normalize the timeout knobs — octal trap defused (Bugbot #442 r6) TB_CREATE_TIMEOUT_MIN / TB_HELM_TIMEOUT_MIN sanitizers accepted 08/09, which bash arithmetic reads as invalid octal — aborting $(( … )) under set -e mid-create (k3d already backgrounded → partial cluster left behind), and 010 silently became 8. One shared, unit-tested helper (tb_minutes_or) now normalizes via 10# at all three call sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* docs(rfc): least-privilege install (0001) — for discussion Draft RFC: the installer assumes blanket root/sudo, which excludes non-admin users (hospital/university/HPC researchers). Audit shows the ONLY privileged surface is the container runtime + two kernel modules; tools + the cluster are already user-space. Proposes a tiered model (zero-root when a runtime exists → rootless → one scoped privileged step) and ties in A2 (sudo/root) + B2 (PATH). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * RFC 0001: record decisions (Docker-only, kernel=Tier-2, build prepare-host + host audit, WSL2=Linux, detection yes) Resolves the five open questions from the first draft per Lukas 2026-07-22: Docker as the single runtime (rootless Docker at Tier 1); kernel modules a hard Tier-2 requirement with no userspace fallback; build a standalone prepare-host step plus a short host-audit report shared with doctor; WSL2 treated as Linux and native Windows preferring rootless Docker; and yes to side-effect-free detection. Rollout re-sequenced so detection + audit land first (also closing A2/B2). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * RFC 0001: adopt @saadqbal review — rootless Docker as primary target Folds in Asad review of #369. Key flip: rootless Docker is the primary path, not a fallback, which dissolves the kernel-module question (fuse-overlayfs + slirp4netns remove overlay/br_netfilter from the privileged surface). Cant-modprobe now falls to rootless rather than failing; we probe cgroup v2 + unprivileged userns instead. prepare-host shipped as snippet + subcommand; WSL2 prefers rootless over Docker Desktop (licensing); detection uses docker-info + the id/sudo/sudo-n trio. Five open questions collapse to one hands-on spike: validate rootless Docker as the k3d backend across target hosts. Rollout re-sequenced so that spike leads the core build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(rfc): renumber to RFC-CLIENT-0002 and record the accepted status Two fixes to get this RFC out of limbo. **Renumber 0001 -> 0002.** `client` already has a `0001-rootless-spike.md` on develop, so this file collided with it inside its own repo. 0002 is the next free number in this repo. The header now carries the qualified ID `RFC-CLIENT-0002` — RFC numbers are per-repo, so a bare "RFC 0001" names four different documents across the org. **Status: Accepted (2026-07-25).** The design has plainly been accepted in practice: the tracking epic tracebloc/backend#1168 is 6-of-9 children merged, including the foundation, tier routing, sudo handling, Tier 0 and the rootless spike. Leaving the document as a permanent "do not merge" draft left the repo with no record of a decision the team had already acted on. Implementation status stays where it belongs — on the epic, not in this header. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…#415) (#449) The NVIDIA container toolkit setup ran entirely through Log (file-only), so the console sat blank through the 30s WSL probe, the multi-hundred-MB Ubuntu install, and the 180s toolkit install — and on failure printed a vague "set it up manually inside WSL later" with no actual commands. - Add Wait-JobWithProgress: spinner + elapsed/timeout heartbeat while a background job runs, bounded, stops the job on timeout. No GPU sub-step now goes silent for more than ~2s. - Add Show-GpuManualRemedy: copy-pastable install commands (same --tlsv1.2 + connect/max-time floor as the automated path) plus a `tracebloc doctor` follow-up, printed on every timeout/failure branch. - Ubuntu install is now a progress-tracked job instead of a silent `cmd /c ... | Out-Null`. - Visible Info intro (optional; CPU mode works either way) + Ok on success; drop the vague dead-end copy. - Extract shared spinner frames ($script:SpinnerFrames). - 11 Pester tests. Acceptance: no silent window >10s; timeout output carries runnable remedies. Kept in Step 1 (the --gpus flag must be decided before New-K3dCluster; moving it would force cluster recreation, #431). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ge prod image (#454) * fix(chart): pin mysql-client by digest — keep fleets on the 5.7-lineage prod image (backend#723) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(e2e): era-aware --reuse-values pin assertion — the published release now carries the prod pin The path-1 assertion hardcoded the pre-#398 era: it expected NO ingestor pin after a --reuse-values upgrade because the published release's computed values predated images.ingestor.prodDigest. The #383 promotion (2026-07-27) published a release that includes the pin, so replayed computed values now carry it and the assertion fails on every chart-touching PR (first hit: #454). The expectation is now read from the baseline release itself (helm get values --all): pin absent => must not arrive (old behavior); pin present => the SAME digest must be replayed verbatim. Both eras keep asserting the actual limitation: --reuse-values never injects new chart defaults. Path-2's era-stale comment corrected, with a note on the replay-contamination signal a future pin bump will surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(e2e): fail fast when jq is missing (review: local runs, not just CI runners) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(e2e): one era boundary, stated once (the #383 promotion); track the path-2 tripwire in #459 (review) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(chart)+test: single-arch pin note + render assertion for the digest path (review) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ow carries the prod pin (#455) * fix(e2e): era-aware --reuse-values pin assertion — the published release now carries the prod pin The path-1 assertion hardcoded the pre-#398 era: it expected NO ingestor pin after a --reuse-values upgrade because the published release's computed values predated images.ingestor.prodDigest. The #383 promotion (2026-07-27) published a release that includes the pin, so replayed computed values now carry it and the assertion fails on every chart-touching PR (first hit: #454). The expectation is now read from the baseline release itself (helm get values --all): pin absent => must not arrive (old behavior); pin present => the SAME digest must be replayed verbatim. Both eras keep asserting the actual limitation: --reuse-values never injects new chart defaults. Path-2's era-stale comment corrected, with a note on the replay-contamination signal a future pin bump will surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(e2e): fail fast when jq is missing (review: local runs, not just CI runners) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(e2e): one era boundary, stated once (the #383 promotion); track the path-2 tripwire in #459 (review) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
#416) (#450) * feat(installer): preflight probes every download host, cross-OS + hard (#416) Preflight proved connectivity to registries + tracebloc endpoints, then Step 1 downloaded from hosts it never probed — so an all-green preflight was followed ~30s later by a blocked-download failure on TLS-intercepting / allowlist networks, misleading the IT contact who just watched preflight pass. - Probe every download host the default path fetches from, per-OS and HARD (a blocked one is now a named red preflight line, not a warn), but only when the fetch will actually happen (tool/app absent — a present tool is never re-downloaded, so its host isn't probed). * always: auth.docker.io (Docker Hub token host — allowed registry-1 but blocked token host used to fail only at in-cluster pull time) * Linux: get.docker.com, download.docker.com, github.com + objects.githubusercontent.com, dl.k8s.io, get.helm.sh * macOS: raw.githubusercontent.com (Homebrew), desktop.docker.com * Windows: desktop.docker.com, dl.k8s.io, get.helm.sh, github.com + objects.githubusercontent.com - objects.githubusercontent.com is probed explicitly: release assets 302 there and _pf_probe_url does not follow redirects, so github.com passing proved nothing about the asset host. - GPU hosts are deliberately excluded (GPU setup is optional; #415 handles its failure with runnable remedies — hard-failing preflight would contradict that). - New check-drift.sh parity check (_drift_preflight_hosts) locks the shared-core host set so the two installers can't drift apart. Fixes both halves in lockstep (preflight.sh + install-k8s.ps1). Tests: +5 bats (preflight), +2 bats (drift), +7 Pester; the one existing warn-only assertion (preflight.bats) is updated to the new hard behaviour. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): Docker-engine host is warn, not hard; egress hint names all hosts (Bugbot #416) Bugbot found the download-host hard-probing over-generalized which host each path fetches Docker from, so a blocked but UNUSED host could abort a supported install: - macOS: desktop.docker.com hard-failed headless Macs that use Colima via brew. - Linux: get.docker.com / download.docker.com both hard, but pacman/zypper/Amazon use distro repos and RHEL clones use only download.docker.com. The Docker-engine install host is path/distro/environment-dependent, so it's now WARN-only (soft bucket) on Linux + macOS. The k8s tool binaries (dl.k8s.io, get.helm.sh, github.com + objects.githubusercontent.com) stay HARD — they're always direct-downloaded from the same host. Windows is unchanged: Docker Desktop is its sole path, so install-k8s.ps1 keeps desktop.docker.com hard. Also (Bugbot medium): the egress hint listed only the old always-critical hosts; it now names the tool-download hosts too, so a red line for a blocked download host has matching remediation. Mirrored in preflight.sh + install-k8s.ps1. Updated the two #416 bats tests that asserted the old hard behavior to assert warn-only for the Docker-engine host. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): macOS also probes github.com for the Homebrew clone (Bugbot #416) install_homebrew fetches the install script from raw.githubusercontent.com and then git-clones Homebrew/brew + core from github.com. Preflight probed only the raw host, so a network that allows it but blocks github.com passed preflight then failed during Homebrew setup — the same multi-host gap already closed for k3d via objects.githubusercontent.com. Probe github.com too on macOS when brew is absent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): probe formulae.brew.sh; drift check extracts probe URLs (review #416) Two reviewer findings (saadqbal): 1. formulae.brew.sh unprobed on macOS: `brew install` pulls formula METADATA from formulae.brew.sh (bottles come from ghcr.io, already probed), and it's hit even when brew is already installed. A blocked metadata host = green preflight then a failed `brew install`. Now hard-probed on macOS whenever a brew-installed tool (kubectl/k3d/helm/docker) is absent. 2. Drift check couldn't detect a deleted probe: it grepped the whole file, so each shared host also matched inside comments and the egress-hint strings — passing even if a real probe line were removed (the AC wasn't enforced; the bats cases only passed because the fixtures lacked that text). check-drift now extracts only the hosts in an actual PROBE URL (bash "…|https://host/…", ps1 url = "https://host/…") and diffs those. tracebloc.github.io drops from the shared set (ps1 probes it via $TRACEBLOC_HELM_REPO_URL, not a literal; Check 1 already pins the host map). Rewrote the drift bats fixtures to real probe entries + added a case proving a comment/hint-only host is still flagged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): tighten drift host extraction + formulae.brew.sh trigger (Bugbot #416) Two Bugbot findings, both refining the earlier reviewer fixes (not reverting them): - Drift check (ps1 side) matched ANY `url = "https://…"`, including the winget bootstrap download line, so deleting the real k3d github.com probe still passed. Scope the extractor to hashtable probe entries (require `label =` on the line), so only genuine preflight probes count. Added a bats case with a stray $url= download line + a deleted probe -> now correctly flags drift. - formulae.brew.sh was hard-probed on `! has docker` too, but GUI Macs install Docker Desktop (desktop.docker.com), not brew — so docker-only-missing would hard-fail a host the install never uses. Trigger only on kubectl/k3d/helm (the tools that always install via brew). Added a bats case for docker-only-missing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): probe formulae.brew.sh for headless (Colima) docker installs too (Bugbot #416) Round-3 dropped docker from the formulae.brew.sh trigger to avoid a GUI-Mac false-fail (Docker Desktop, not brew). But headless Macs install colima/docker via `brew install`, which DOES hit formulae.brew.sh — so docker-only-missing on a headless box went green in preflight then failed in Step 1. Make it path-aware: add a _pf_has_gui_session helper (mirrors setup-macos.sh) and probe formulae.brew.sh for a missing docker only when there's NO GUI session (the Colima/brew path). GUI Macs still skip it (Docker Desktop). Added bats cases for both GUI and headless. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(installer): consolidate macOS Docker-path egress probing (Bugbot #416) Fold the r3/r4/r5 point-fixes into one GUI-aware block keyed on _pf_has_gui_session, so each host is hard-probed only on the path that fetches it: - kubectl/k3d/helm absent -> formulae.brew.sh (always brew). - docker absent + GUI -> desktop.docker.com HARD (the actual Docker Desktop path; was warn-only -> a blocked CDN passed preflight then failed mid-download, r5). - docker absent + headless -> formulae.brew.sh (colima/docker via brew, r4). desktop.docker.com leaves the warn bucket entirely (it's hard on GUI, unprobed on headless). No behaviour is looser than before; the GUI Desktop CDN is now caught. Tests refreshed: GUI vs headless for both desktop.docker.com and formulae.brew.sh. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
… Step 1 (#411) (#451) * fix(installer): execute-gate installed tools so a broken binary fails Step 1 (#411) The only post-install "verification" was a log-only, failure-masking interpolation, so a corrupt or wrong-architecture binary (winget shims / partial installs skip checksum verify; brew delivers with no checksum of ours) still printed "System tools" and only died at cluster-create in Step 2. All three OSes had the gap. - New shared execute-gate helpers: assert_tool_runs (common.sh, bash) and Assert-ToolRuns (install-k8s.ps1, PowerShell). Each runs the tool's self-check; on non-zero exit or an exception it removes the located binary and fails loudly with an arch-aware remedy, so the tool step fails instead of the cluster step. - Gate kubectl / k3d / helm after install on Linux (setup-linux.sh), macOS (setup-macos.sh), and Windows (install-k8s.ps1), on both the fresh-install and already-present paths. "System tools" success only prints once all gates pass. - kubectl is gated with `version --client` (NOT --short — removed in kubectl 1.28+, which would false-fail the gate). - New check-drift.sh parity check (_drift_execute_gates) so no installer can silently drop a gate for a tool. Tests: +2 bats (common — working tool passes / broken tool errors + removes the binary), +2 bats (drift parity), +7 Pester (exit-nonzero / exception / binary-removal / arch remedy / static gates). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(installer): make setup-linux tool mocks runnable for the execute-gate (#411) The execute-gate (#411) runs `<tool> version` after install, but the setup-linux.bats harness only marked tools "present" via has() — it never provided a RUNNABLE k3d/kubectl/helm. On the toolless CI runner the gate's probe found no binary and failed 4 install_k3d tests; locally it false-passed because a real k3d was on PATH. Add silent (non-recording) runnable stubs to setup() so the gate has something to execute and they shadow any real tool on a dev host. Test-only; no production change. * fix(installer): gate only removes binaries we placed; helm bare version; drift extracts calls (review #411) Three review findings on #411 (saadqbal + Bugbot): - Execute-gate deleted the wrong binary on the already-present path. Removal is now OPT-IN via `assert_tool_runs --rm <path>` / PS `-BinPath`, passed ONLY by the fresh-install callers that placed the binary. On the present / brew / winget path we don't pass it, so a broken pkg-managed binary is left in place (deleting a brew symlink just wedges the re-run, and the bad copy may be elsewhere on PATH). bash: --rm on Linux fresh-install, none on present or macOS/brew. ps1: -BinPath tracks the direct-download dest ($null for winget/present). - helm was gated with `version --short`; --short can be dropped like kubectl's was, which would false-fail the gate and (previously) delete a good binary. Use bare `helm version` on all three OSes. - _drift_execute_gates used a whole-file grep that could match comments. It now strips comment lines and matches the actual call (`assert_tool_runs … <t> version` / `Assert-ToolRuns -Name "<t>"`), handling the new --rm form. (no grep -q under pipefail — SIGPIPE would false-fail.) Tests: common.bats split into --rm-removes vs no-rm-leaves; check-drift.bats adds --rm-form + commented-out-gate cases. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): execute-gate removes a broken binary only if WE placed it AND it ran (Bugbot #411 r2) Round-1 made removal opt-in but only on the fresh-install path, so a broken installer-placed kubectl/k3d on the ALREADY-PRESENT path (left by a prior run) stayed → `has` true → re-run couldn't self-heal (Bugbot). helm already handled it. Unify all three: callers pass --rm "$TB_TOOLS_DIR/<tool>" (bash) / -BinPath "$TOOL_DIR\<tool>.exe" (ps1) on EVERY path, and the gate removes that path only when the binary that actually ran resolves to it — `command -v … -ef <path>` on bash, `(Get-Command).Source -eq $BinPath` on ps1. So a broken copy we own (fresh or prior-run) self-heals, while a brew/winget/pkg-manager copy elsewhere on PATH is never touched (satisfies the earlier reviewer guard too). Dropped the now-moot $k3dDest/$helmDest null-tracking and helm's -f branch. Tests: common.bats +decoy-copy case; install-k8s.Tests.ps1 removal split into ran-here (removed) vs resolved-elsewhere (left); check-drift already covers --rm. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
… train) (#460) * ci(release): tag/chart-version guard + strict stability rule (backend#1301 Q5) The tag's base X.Y.Z must equal client/Chart.yaml's version (train-cut or manual), and any non-plain-semver tag must be a PRE-release -- otherwise it would become 'latest' (the installer bootstrap) and enter the helm index as stable. Tag passed via env per R8 (backend#889). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(release): guard gates BOTH jobs + self-healing demotion (Bugbot) The guard lived only inside the release job; sign-installer-manifest has no needs and kept stamping + attaching installers for a bad release -- which, left marked stable, already IS 'latest' (what the bootstrap resolves). Restructured: a dedicated verify job gates both jobs. And instead of fail-and-strand, an unmarked non-final tag is DEMOTED to prerelease ('latest' snaps back to the previous stable) and publishing continues as a proper pre-release. Chart-version mismatch stays a hard fail -- nothing to auto-fix without a bump. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(release): demote before hard-fail (Bugbot round 2) A release that is BOTH mis-marked stable AND chart-version-mismatched hit the exit 1 before the demotion ran, stranding it as 'latest' with no assets (404ing bootstrap). Demotion now runs first -- it is safe in isolation and must not be skippable by any other failure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
LukasWodka
commented
Jul 29, 2026
ContributorAuthor
bugbot run |
Uh oh!
There was an error while loading. Please reload this page.
LukasWodka
commented
Jul 29, 2026
ContributorAuthor
bugbot run |
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 6bbc9f3. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated promotion by the release train (RFC-0008 D14). Head is the train-managed
release-train/to-stagingbranch (a mirror ofdevelop), so it never collides with a human PR. Merged only when the fr-gate is green.Note
High Risk
Fleet-wide prod ingestor pinning and removal of the metadata-backfill hook affect every prod edge on upgrade; combined with release demotion logic and large Windows installer behavior changes, regressions would show up as failed ingestion, stuck installs, or bad releases marked latest.
Overview
Helm client 1.9.7 promotes prod ingestor reproducibility by putting the fleet pin in chart defaults (
images.ingestor.prodDigest/prodPin, resolved viatracebloc.ingestorDigestforINGESTOR_IMAGE_DIGEST) and removingclient/values-prod.yaml. Dev/staging keep floating onimages.ingestor.tagwhenCLIENT_ENVisdev/stg. CIingestor-multiarchnow requires a non-emptyprodDigestand validates it as multi-arch.mysql-clientgets a default digest pin; the metadata backfill post-upgrade hook, its tests, andmetadataBackfillvalues/schema are removed.Release workflow adds a shared
verifyjob (tag vsChart.yamlversion; demote non-final tags published as stable) before chart publish and installer manifest signing.Installers: Bash gains
prepare-hostrouting and an early network-FS data-dir guard;check-style.shflags barecurl.install-k8s.ps1is reordered (CLI → browser provisioning → Helm), adds Windows leftover-data guard, bounded k3d/Docker waits, pinned k3d/helm downloads, tool execute-gates, expanded preflight egress checks, and adopted-client Helm reconcile. Docs/PR template/Bugbot guide and RFC renumbering (RFC-CLIENT-0001/0002) ship with the train.Reviewed by Cursor Bugbot for commit 6bbc9f3. Bugbot is set up for automated code reviews on this repo. Configure here.