Uh oh!
There was an error while loading. Please reload this page.
Conversation
Changes in values.yaml according to the test deployment.
… added new entry in index.yaml
…in index.yaml, and adjust Chart.yaml version accordingly.
…ponding appVersion updates and new init-container.yaml for directory creation. Remove old versions and adjust index.yaml entries accordingly.
…ponding appVersion updates to 1.24.6. Remove old versions and adjust index.yaml entries accordingly. Add new ConfigMap for MySQL configuration in deployment templates.
…reated timestamps and digests in index.yaml. Comment out log-error in MySQL client deployment template.
…oyment.yaml and update created timestamps and digests in index.yaml for aks, bm, and eks Helm charts.
…aks and bm, and refresh created timestamps and digests in index.yaml for all Helm charts.
latest->dev
latest->dev
saadqbal pushed a commit
that referenced
this pull request
Jun 2, 2026
A pre-merge correctness review (high effort) over the stack found three real
bugs — the failure paths weren't exercised by the passing tests:
1. cluster.sh: `k3d "${K3D_ARGS[@]}" > ...` was a bare command under `set -e`,
so a k3d-create FAILURE aborted the script immediately, skipping the
'already exists' graceful reuse, the error dump, AND the proxy temp-dir
cleanup. Capture rc set-e-safely (`&& create_rc=0 || create_rc=$?`).
Proven under set -e: both the error-dump and reuse paths now run.
2. summary.sh: CLIENT_STATE was defaulted to "starting" at source time, so
install_cleanup's `[[ -z "$CLIENT_STATE" ]]` guard was always false and the
"did not complete / check the log / safe to re-run" hint never printed on an
early failure (preflight / docker / cluster / helm). Default it empty; the
readiness gate sets the real state.
3. preflight.sh: with curl absent (direct ./install-k8s.sh on a minimal VM,
before install_system_deps adds curl), the connectivity probes returned
'nocurl' and hard-failed with a misleading "egress blocked". Skip the check
with a warning when curl isn't present yet.
Also defensively quote `switch ("$env:CLIENT_ENV")` in Get-BackendUrl so the
prod default fires regardless of PowerShell version (refuted as a live bug on
pwsh 7 -- default does fire -- but cheap insurance for the unvalidated 5.1 path).
Tests: +nocurl-skip test, fixed an over-blunt has() mock; bats 120 / Pester 52
green. (#1's set -e abort can't be caught by bats -- no set -e there -- so it
was verified manually under set -e.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>saadqbal added a commit
that referenced
this pull request
Jun 2, 2026
* feat(installer): --diagnose support bundle (redacted) Adds a one-command support bundle so a customer hitting an install/runtime problem can send a single file instead of a multi-round log-gathering email thread (the Charité thread is the archetype). `bash <(curl ... i.sh) --diagnose` (or `install-k8s.ps1 -Diagnose`) collects logs + cluster/host status into ~/.tracebloc/tracebloc-diagnose-<ts>.tgz. Two guarantees: - Best-effort: the whole collection runs under `set +e` and short-circuits before any install work, so it works even when the install is broken. - Credential-safe: clientPassword, proxy credentials (user:pass@host), and password=/token/secret values are REDACTED from every file before archiving; clientId is kept (it's the identifier support needs, not a secret). scripts/lib/diagnose.sh -- run_diagnose() (host/versions, docker/k3d, kubectl overview + describe of non-Running pods, workload logs with namespace auto-discovery, helm, install log + values.yaml, proxy env) + _redact_file(). install-k8s.sh sources it + adds the --diagnose short-circuit (clears the EXIT trap so the post-install message doesn't fire); install.sh adds it to the bootstrap download manifest. install-k8s.ps1 mirrors with -Diagnose / Invoke-DiagnoseBundle / Edit-Redaction. Documented in --help. Tests: diagnose.bats (7, incl. the end-to-end redaction gate) + Pester (+3). Verified on a Linux VM: the real --diagnose flag produced a 16-file bundle and the seeded dev password + proxy credentials had ZERO occurrences in the archive (clientId kept); also works with no cluster present. Stacked on #171 + #172 + #173 + #174. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(installer): raise changed-line coverage (diagnose collection + real preflight probes) Measured changed-line coverage of the stack had dropped (bash ~84% vs #171's ~96%) because the new code added integration-only branches the mocked unit suites skipped. Recover the unit-testable portion: - diagnose.bats: exercise the kubectl/docker/helm collection path (has()=true + mocked tools) -> diagnose.sh 64% -> 90%. - preflight.bats: test the REAL _pf_probe_url curl-exit-code -> token mapping, the missing-curl path, and the _pf_ncpu/_pf_total_mem_kb/_pf_free_kb readers (re-sourced past the setup stubs) -> preflight.sh 79% -> 90%. Bash changed-line coverage: 83.6% -> 92.3% (kcov, 383/415). The residual ~8% is integration-only (real k3d/docker create + macOS/Windows-specific branches + MAIN orchestration), validated by the live VM E2Es (reboot recovery, auth-proxy, preflight blocked-egress/arm64, diagnose redaction grep). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): close two redaction gaps in --diagnose (security review) A pre-merge security review of the support bundle found two ways credentials could land in the (supposedly redacted) archive the customer sends to support: 1. Redaction only matched `clientPassword:` and `password=` -- it missed any other *password key in colon form, so `dockerRegistry.password` (a registry token) and `HTTP_PROXY_PASSWORD` survived. Broadened _redact_file (bash) and Edit-Redaction (ps1) to redact ANY *password key, case-insensitive, in : or = form (portable explicit char classes -- BSD sed has no I flag). 2. (bash only) The bundle collected `helm get manifest`, which renders the k8s Secret objects with base64-encoded CLIENT_PASSWORD + .dockerconfigjson that text redaction can't see. Dropped the manifest collection (helm get values + kubectl output already cover triage). Regression tests added (diagnose.bats + Pester). Re-verified end-to-end on a Linux VM with the real --diagnose flag: clientPassword, the dockerRegistry token, HTTP_PROXY_PASSWORD, and proxy URL creds all have ZERO occurrences in the archive; clientId kept; manifest no longer collected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): correctness fixes from code review A pre-merge correctness review (high effort) over the stack found three real bugs — the failure paths weren't exercised by the passing tests: 1. cluster.sh: `k3d "${K3D_ARGS[@]}" > ...` was a bare command under `set -e`, so a k3d-create FAILURE aborted the script immediately, skipping the 'already exists' graceful reuse, the error dump, AND the proxy temp-dir cleanup. Capture rc set-e-safely (`&& create_rc=0 || create_rc=$?`). Proven under set -e: both the error-dump and reuse paths now run. 2. summary.sh: CLIENT_STATE was defaulted to "starting" at source time, so install_cleanup's `[[ -z "$CLIENT_STATE" ]]` guard was always false and the "did not complete / check the log / safe to re-run" hint never printed on an early failure (preflight / docker / cluster / helm). Default it empty; the readiness gate sets the real state. 3. preflight.sh: with curl absent (direct ./install-k8s.sh on a minimal VM, before install_system_deps adds curl), the connectivity probes returned 'nocurl' and hard-failed with a misleading "egress blocked". Skip the check with a warning when curl isn't present yet. Also defensively quote `switch ("$env:CLIENT_ENV")` in Get-BackendUrl so the prod default fires regardless of PowerShell version (refuted as a live bug on pwsh 7 -- default does fire -- but cheap insurance for the unvalidated 5.1 path). Tests: +nocurl-skip test, fixed an over-blunt has() mock; bats 120 / Pester 52 green. (#1's set -e abort can't be caught by bats -- no set -e there -- so it was verified manually under set -e.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): handle a Docker daemon that won't start (stop misdiagnosing it as a group issue) Asad's AlmaLinux 9 / EC2 test: docker-ce installed fine but `dockerd` crashed on startup (exit 1), systemd throttled it ("Start request repeated too quickly"), and the installer then printed "Could not connect to Docker -- try logging out and back in" -- the GROUP-not-active hint, which is wrong for a dead daemon and sent him in circles (logout/login didn't help). The throttle also means a bare re-run can't recover. install_docker_engine now: - `systemctl enable docker` WITHOUT `--now` (a start failure no longer hard-aborts the script under `set -e` at that line); - `systemctl reset-failed docker` before starting, so a throttled/failed unit from a prior attempt can be retried (a plain re-run now recovers); - when `docker info` fails AND the daemon isn't active (vs. the group-not-active case, which is still re-exec'd via `sg docker`), surface Docker's OWN error (systemctl status + the journalctl error lines) with likely RHEL/AlmaLinux causes, instead of the misleading group hint. Test: setup-linux.bats daemon-won't-start case; bats 121 green. NOTE: this fixes the installer's HANDLING. Asad's root cause (why dockerd exits 1 on that box) is still masked by the systemd throttle and is being chased separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): load the kernel modules Docker/k3s need (fixes dockerd on minimal RHEL/AlmaLinux) Root cause of Asad's AlmaLinux 9 / EC2 failure: dockerd died on startup with "failed to register bridge driver: iptables ... addrtype ... missing kernel module". Minimal RHEL/AlmaLinux cloud images (incl. AWS) ship kernel-modules-core but NOT the full kernel-modules package, so xt_addrtype (+ br_netfilter, overlay) aren't available and Docker can't program its bridge NAT rules. New _ensure_kernel_modules() (setup-linux.sh), called before starting Docker: modprobe overlay / br_netfilter / xt_addrtype / iptable_nat / ip_tables; on RHEL-family, if a load fails, `dnf install kernel-modules-$(uname -r)` and retry; persist to /etc/modules-load.d for reboots. Best-effort + idempotent (verified clean on a healthy Ubuntu box). Also sharpened the daemon-won't-start hint to point at the kernel-modules remedy when the error mentions addrtype/missing module. This is the hospital-VM profile (minimal RHEL/Alma), so it's a real install-side fix, not just error handling. Test: setup-linux.bats _ensure_kernel_modules; bats 122. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(installer): cross-distro prereq matrix + real-Windows Pester + static gate Adds .github/workflows/installer-tests.yaml to validate the installer across the breadth of environments customers actually run — not just Ubuntu-amd64: • static — shellcheck (clean at --severity=warning) + bash -n + PSScriptAnalyzer • unit-bash — bats (mocked), 124 tests • unit-pester — Pester on Linux pwsh AND real windows-latest (the .ps1's true target) • distro-prereqs — NEW: runs the REAL Linux prereq path (PM detect, system deps, Docker branch, kernel modules, kubectl/k3d/helm) in a fresh container per distro family: ubuntu 22.04/24.04, debian 12, almalinux 9/8, rockylinux 9, amazonlinux 2023, fedora, opensuse leap. The matrix paid for itself before it even shipped: validating it locally against real distro containers surfaced a genuine gap — minimal Amazon Linux 2023 ships no openssl/tar, so helm's get-helm-3 fails ("openssl must first be installed"). Fixed in install_system_deps (ensure openssl + tar; package names are uniform across apt/dnf/yum/zypper/pacman), with bats coverage. All 9 validated distro branches now install every prerequisite. Installer-test jobs moved out of helm-ci.yaml into their own workflow (no more duplicate runs; helm-ci no longer triggers on scripts/** changes). Arch omitted (x86-only image + bare-container keyring friction; pacman branch covered by bats). Real k3d cluster-up (e2e) intentionally deferred — needs a stubbed backend; tracked. Validated locally via mac Docker: ubuntu:22.04, almalinux:9, amazonlinux:2023, opensuse/leap:15.6 → all PASS; bats 124 green; shellcheck 0 findings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(installer): fix static gate (.bats ≠ bash) + Windows-safe Confirm-Config test The new workflow's first run caught two issues — exactly its job: 1. Static analysis failed: `bash -n` ran on .bats files, which are bats DSL (@test "name" { … }), not valid bash. Restrict the syntax check to *.sh; .bats are validated by being run in the unit-bash job. 2. Pester on real windows-latest failed 1/55: the Confirm-Config test set $env:USERPROFILE = $env:HOME, but $env:HOME is empty on Windows, so [System.IO.Path]::GetFullPath("") threw "path is empty". The INSTALLER is correct (defaults to $env:USERPROFILE, always set on Windows) — the test fixture was Linux-centric. Derive a profile dir valid on both OSes. For the record, the first run's wins: all 9 distro prereq jobs (ubuntu 22.04/ 24.04, debian 12, almalinux 8/9, rockylinux 9, amazonlinux 2023, fedora, opensuse leap) + bats + Linux Pester passed on GHA's amd64 runners. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(installer): gate ShellCheck at error severity (SC2034 cross-file false positives) The libs are sourced together as one program, so single-file shellcheck reports SC2034 "unused" for shared vars defined in common.sh and consumed in other sourced files (CURL_SECURE, ARCH_DL, colours…). Gate at --severity=error (0 findings); warnings still printed as advisory. With this, Static analysis joins the already-green distro matrix + Windows/Linux Pester + bats. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(installer): real k3d cluster-up E2E on Ubuntu (amd64 + arm64) Adds scripts/tests/e2e-cluster.sh + an e2e-cluster matrix job — the highest- fidelity check CI can run. It drives the installer's OWN create_cluster() to bring up an actual k3d cluster on a real kernel (Docker is preinstalled on the runner), asserts every node reaches Ready, then proves the cluster can pull, schedule, and run a public workload (nginx:alpine), and tears down. It deliberately stops BEFORE the tracebloc helm install / backend registration (private images + real credentials), so it needs no secrets. Runs on ubuntu-22.04, ubuntu-24.04, and ubuntu-24.04-arm (arm64 runners are free on this public repo) — covering the real cluster path on both architectures. Validated locally on an arm64 Ubuntu VM: create_cluster() → server+agent Ready (k3s v1.29.4) → nginx pod Running → teardown. shellcheck clean (0 errors). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(installer): fix E2E probe race on the default ServiceAccount The amd64 runners proved the cluster comes up fine (nodes Ready), but the probe pod failed with "serviceaccount default not found" — kubectl run raced the SA controller, which creates default/default asynchronously after the node goes Ready. arm64 dodged it by timing. Wait for the SA before running the pod. Pure test-harness fix; the installer cluster path is correct on all arches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(installer): authenticated corporate-proxy E2E (squid) Adds scripts/tests/e2e-proxy.sh + an e2e-proxy job. Stands up a squid that REQUIRES basic auth, brings up a k3d cluster via the installer's create_cluster() with HTTP(S)_PROXY=http://user:pass@host.k3d.internal:3128, and proves the nodes pull a workload image THROUGH the authed proxy — the squid access log shows an authenticated CONNECT to auth.docker.io (which only a real image pull makes, never the readiness probe), closing the "proxy silently bypassed" false positive. It also asserts anonymous requests are refused, so auth is genuinely enforced. Guards the corporate-proxy hardening end-to-end (#172/#174, the Charité/hospital archetype): _write_k3d_proxy_config passes proxy env via a k3d config FILE so the '@' in user:pass@host survives (k3d splits --env on '@'), plus _augment_no_proxy. If the credentials regress, squid 407s and the pull hangs — the test fails loudly. Stops before the helm install / backend registration; no secrets. Validated locally on an arm64 Ubuntu VM: anonymous refused → cluster up via the authed proxy → nginx pulled through it (auth.docker.io + registry-1.docker.io CONNECTs by the proxy user) → teardown. shellcheck clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): install kernel-modules-extra + handle reboot-required (#176) dockerd crash-loops on minimal RHEL/AlmaLinux images because xt_addrtype/ iptable_nat/br_netfilter live in kernel-modules-extra, not the base kernel-modules package. The prior fix installed kernel-modules-$(uname -r) — the wrong package — so the self-heal never took. Install kernel-modules-extra (unversioned). When the repo's extra modules target a newer kernel than the running one (stale AMI), they can't load until reboot: detect that, set KMODS_REBOOT_REQUIRED, and have install_docker_engine print a clear reboot-and-re-run message instead of a raw Docker error. Modules persist via /etc/modules-load.d/tracebloc.conf. Verified end-to-end on a pristine AlmaLinux 10.1 MINIMAL EC2 box: reboot gate fires, post-reboot modules load, re-run reaches Connected. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Asad Iqbal <asad.dsoft@gmail.com>
saadqbal added a commit
that referenced
this pull request
Jun 2, 2026
* Sync main → develop after v1.4.2 release (#170) * Installer: verify readiness & credentials before reporting success; RHEL-family support (#171) * fix(installer): run on RHEL-family Linux (#718, #719, #720) Docker: install docker-ce from the official Docker CentOS dnf repo on AlmaLinux/Rocky/Oracle, which get.docker.com rejects as unsupported. k3d: preserve PATH through sudo so the post-install lookup survives RHEL secure_path (which omits /usr/local/bin). conntrack: use the conntrack apt package on Debian/Ubuntu and conntrack-tools elsewhere. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): verify credentials and readiness before reporting success (#716, #717) Credentials entered at the prompt are validated against the backend api-token-auth endpoint (the same call jobs-manager makes) with a re-prompt loop, so a wrong Client ID or password is caught immediately instead of after a full deploy. After helm apply, wait_for_client_ready polls rollout status and classifies the outcome; print_summary reports connected, starting, bad_creds, image_pull or crash, and prints the data-never-leaves message only when the client is verifiably connected. Exit code now reflects the real outcome. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): mirror credential and readiness checks on Windows (#716, #717) Test-Credentials, Wait-ForClientReady and Get-NotReadyState mirror the bash logic in install-k8s.ps1, and Print-Summary is now state-branched. Validated with the PowerShell 7.4 parser; runtime behavior still needs a check on a Windows host. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(installer): bats + Pester unit suites, wired into CI scripts/tests/: 64 bats tests (summary, install-client-helm, setup-linux, common) + 32 Pester tests for install-k8s.ps1 — all mocked, no Docker/k3d/network needed. Changed-line coverage measured with kcov (bash 96.2%) and Pester (PowerShell 97.4%); residual lines are the real RHEL Docker-install commands + the guarded main() orchestration, exercised by the integration E2E. A TB_PESTER guard lets the suite dot-source install-k8s.ps1 without running the installer. New installer-tests job in helm-ci.yaml runs both suites on PRs (scripts/ added to path filters). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): harden corporate-proxy support (auth proxy, NO_PROXY, 0.0.0.0 detect, Windows parity) (#172) A customer running behind an authenticated corporate HTTP proxy hit install failures. The 0.0.0.0 kubeconfig headline was fixed for the bash path in #166/#167, but adverse testing (a forward proxy + real k3d on Linux VMs) surfaced three remaining gaps plus a Windows parity hole: - Gap A: authenticated proxies (http://user:pass@host) were silently SKIPPED — k3d's --env KEY=VALUE@FILTER can't carry an '@' in the value. Now propagated via a k3d --config file (structured YAML env) so credentials survive intact. Verified on k3d v5.8.3 (it merges the --config env with the existing CLI flags). - Gap B: NO_PROXY was propagated verbatim. Now auto-augmented with the cluster-internal ranges (loopback + RFC1918 + .svc/.cluster.local + host.k3d.internal), both into the cluster and host-side, so in-cluster traffic never routes through the proxy — fixes the misroute AND the observed `k3d cluster create --wait` hang. - Gap C: a cluster created outside the installer and bound to 0.0.0.0 is now detected (serverlb HostIp) and flagged with a non-destructive recreate remedy. - Windows parity: install-k8s.ps1::New-K3dCluster had NONE of the bash fixes — it still bound --api-port 0.0.0.0:6550 (the original headline bug, still live on Windows), normalized only host.docker.internal in the kubeconfig, and propagated zero proxy env. Now mirrors bash: 127.0.0.1:6550, a 0.0.0.0->127.0.0.1 kubeconfig rewrite, and Get-EffectiveNoProxy + Write-K3dProxyConfig (auth + augmented NO_PROXY, written UTF-8 without a BOM). Tests: new scripts/tests/cluster.bats (15) + Pester for the two ps1 helpers (6), both green. Verified end-to-end on Linux VMs: auth creds propagated into the node, no startup hang behind an unreachable proxy, and 0.0.0.0 detection firing. Stacked on #171 (the installer test scaffolding + final install-k8s.ps1 live there). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): preflight gate (arch, egress, disk, RAM, CPU) — fail fast, clearly (#173) Most install failures fall into two shapes: the environment can't support what the installer does and it fails CRYPTICALLY minutes in, or it claims success it hasn't earned. #171/#172 fixed specific cases; this attacks the first pattern systematically with a preflight gate that runs before any install/cluster work and fails in seconds with a precise, actionable reason. scripts/lib/preflight.sh — run_preflight() runs at the top of Step 1/4: - Architecture (arm64 guard): the tracebloc client images (e.g. mysql-client) are amd64-only. amd64 -> ok; arm64 on Docker Desktop -> info (emulated); arm64 Linux without QEMU binfmt -> hard fail with the `tonistiigi/binfmt --install amd64` remedy. Override: TRACEBLOC_ALLOW_ARM64=1. (This is exactly the `exec format error` we hit on arm64.) - Egress connectivity: probes the endpoints the install needs and reports which are blocked. Hard-fail on the criticals (registry-1.docker.io, ghcr.io, the CLIENT_ENV backend, tracebloc.github.io); warn-only on tool-download hosts when the tool isn't already installed. A TLS/cert error emits a break-and-inspect proxy hint. The probe honors HTTP_PROXY. - Disk / RAM / CPU: hard-fail on critically low disk; warn on low disk/RAM/CPU. - Aggregates results (runs ALL checks, then exits once with a summary). Escape hatch: TRACEBLOC_SKIP_PREFLIGHT=1. install-k8s.sh sources + calls run_preflight; install.sh adds preflight.sh to the curl-bootstrap download manifest (verified: every sourced lib is downloaded). install-k8s.ps1: Test-Preflight + Get-Pf* helpers mirror the bash logic for Windows (Get-CimInstance disk/RAM/CPU, Invoke-WebRequest connectivity). docs/INSTALL.md: a "Network requirements (egress allowlist)" section the preflight error points users to. Tests: scripts/tests/preflight.bats (21) + Pester Describes (Test-PfUrl, Test-Preflight; the Get-Cim* readers are Windows-only so they skip off-Windows). Verified end-to-end on a Linux VM: healthy run passes; a blocked ghcr.io fails in ~1s naming the host; arm64 without binfmt fails with the remedy. Stacked on #171 + #172. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): guarantee reboot persistence + make it visible (#174) A real VM reboot test showed the k3d cluster already auto-recovers on Linux (k3d sets --restart unless-stopped on its nodes; the Docker install enables docker.service on boot). So no systemd unit is needed — this GUARANTEES that behavior even on edge setups, and tells the user about it: - cluster.sh: new ensure_cluster_autostart() (called from create_cluster) -- `docker update --restart unless-stopped` on the k3d nodes (covers externally-created clusters / a future k3d default change) + `systemctl enable docker` on Linux (covers the installed-but-disabled re-run case the fresh-install path misses). Opt out with TRACEBLOC_NO_AUTOSTART=1. - summary.sh: _reboot_note() in the connected summary -- Linux: "Survives reboot"; macOS/Windows: enable Docker Desktop start-on-login. - install-k8s.ps1: Set-ClusterAutostart (defensive unless-stopped on the nodes) + the Docker Desktop reboot note in Print-Summary. Tests: cluster.bats (+4), summary.bats (+3), Pester (+2) -- all green. Verified by a real reboot on a Linux VM: with the restart policy stripped (policy=no), ensure_cluster_autostart restored it to unless-stopped + enabled docker; after `limactl stop/start`, the cluster, node, AND a deployed workload all returned Running with no intervention. Stacked on #171 + #172 + #173. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): --diagnose support bundle (redacted) (#175) * feat(installer): --diagnose support bundle (redacted) Adds a one-command support bundle so a customer hitting an install/runtime problem can send a single file instead of a multi-round log-gathering email thread (the Charité thread is the archetype). `bash <(curl ... i.sh) --diagnose` (or `install-k8s.ps1 -Diagnose`) collects logs + cluster/host status into ~/.tracebloc/tracebloc-diagnose-<ts>.tgz. Two guarantees: - Best-effort: the whole collection runs under `set +e` and short-circuits before any install work, so it works even when the install is broken. - Credential-safe: clientPassword, proxy credentials (user:pass@host), and password=/token/secret values are REDACTED from every file before archiving; clientId is kept (it's the identifier support needs, not a secret). scripts/lib/diagnose.sh -- run_diagnose() (host/versions, docker/k3d, kubectl overview + describe of non-Running pods, workload logs with namespace auto-discovery, helm, install log + values.yaml, proxy env) + _redact_file(). install-k8s.sh sources it + adds the --diagnose short-circuit (clears the EXIT trap so the post-install message doesn't fire); install.sh adds it to the bootstrap download manifest. install-k8s.ps1 mirrors with -Diagnose / Invoke-DiagnoseBundle / Edit-Redaction. Documented in --help. Tests: diagnose.bats (7, incl. the end-to-end redaction gate) + Pester (+3). Verified on a Linux VM: the real --diagnose flag produced a 16-file bundle and the seeded dev password + proxy credentials had ZERO occurrences in the archive (clientId kept); also works with no cluster present. Stacked on #171 + #172 + #173 + #174. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(installer): raise changed-line coverage (diagnose collection + real preflight probes) Measured changed-line coverage of the stack had dropped (bash ~84% vs #171's ~96%) because the new code added integration-only branches the mocked unit suites skipped. Recover the unit-testable portion: - diagnose.bats: exercise the kubectl/docker/helm collection path (has()=true + mocked tools) -> diagnose.sh 64% -> 90%. - preflight.bats: test the REAL _pf_probe_url curl-exit-code -> token mapping, the missing-curl path, and the _pf_ncpu/_pf_total_mem_kb/_pf_free_kb readers (re-sourced past the setup stubs) -> preflight.sh 79% -> 90%. Bash changed-line coverage: 83.6% -> 92.3% (kcov, 383/415). The residual ~8% is integration-only (real k3d/docker create + macOS/Windows-specific branches + MAIN orchestration), validated by the live VM E2Es (reboot recovery, auth-proxy, preflight blocked-egress/arm64, diagnose redaction grep). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): close two redaction gaps in --diagnose (security review) A pre-merge security review of the support bundle found two ways credentials could land in the (supposedly redacted) archive the customer sends to support: 1. Redaction only matched `clientPassword:` and `password=` -- it missed any other *password key in colon form, so `dockerRegistry.password` (a registry token) and `HTTP_PROXY_PASSWORD` survived. Broadened _redact_file (bash) and Edit-Redaction (ps1) to redact ANY *password key, case-insensitive, in : or = form (portable explicit char classes -- BSD sed has no I flag). 2. (bash only) The bundle collected `helm get manifest`, which renders the k8s Secret objects with base64-encoded CLIENT_PASSWORD + .dockerconfigjson that text redaction can't see. Dropped the manifest collection (helm get values + kubectl output already cover triage). Regression tests added (diagnose.bats + Pester). Re-verified end-to-end on a Linux VM with the real --diagnose flag: clientPassword, the dockerRegistry token, HTTP_PROXY_PASSWORD, and proxy URL creds all have ZERO occurrences in the archive; clientId kept; manifest no longer collected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): correctness fixes from code review A pre-merge correctness review (high effort) over the stack found three real bugs — the failure paths weren't exercised by the passing tests: 1. cluster.sh: `k3d "${K3D_ARGS[@]}" > ...` was a bare command under `set -e`, so a k3d-create FAILURE aborted the script immediately, skipping the 'already exists' graceful reuse, the error dump, AND the proxy temp-dir cleanup. Capture rc set-e-safely (`&& create_rc=0 || create_rc=$?`). Proven under set -e: both the error-dump and reuse paths now run. 2. summary.sh: CLIENT_STATE was defaulted to "starting" at source time, so install_cleanup's `[[ -z "$CLIENT_STATE" ]]` guard was always false and the "did not complete / check the log / safe to re-run" hint never printed on an early failure (preflight / docker / cluster / helm). Default it empty; the readiness gate sets the real state. 3. preflight.sh: with curl absent (direct ./install-k8s.sh on a minimal VM, before install_system_deps adds curl), the connectivity probes returned 'nocurl' and hard-failed with a misleading "egress blocked". Skip the check with a warning when curl isn't present yet. Also defensively quote `switch ("$env:CLIENT_ENV")` in Get-BackendUrl so the prod default fires regardless of PowerShell version (refuted as a live bug on pwsh 7 -- default does fire -- but cheap insurance for the unvalidated 5.1 path). Tests: +nocurl-skip test, fixed an over-blunt has() mock; bats 120 / Pester 52 green. (#1's set -e abort can't be caught by bats -- no set -e there -- so it was verified manually under set -e.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): handle a Docker daemon that won't start (stop misdiagnosing it as a group issue) Asad's AlmaLinux 9 / EC2 test: docker-ce installed fine but `dockerd` crashed on startup (exit 1), systemd throttled it ("Start request repeated too quickly"), and the installer then printed "Could not connect to Docker -- try logging out and back in" -- the GROUP-not-active hint, which is wrong for a dead daemon and sent him in circles (logout/login didn't help). The throttle also means a bare re-run can't recover. install_docker_engine now: - `systemctl enable docker` WITHOUT `--now` (a start failure no longer hard-aborts the script under `set -e` at that line); - `systemctl reset-failed docker` before starting, so a throttled/failed unit from a prior attempt can be retried (a plain re-run now recovers); - when `docker info` fails AND the daemon isn't active (vs. the group-not-active case, which is still re-exec'd via `sg docker`), surface Docker's OWN error (systemctl status + the journalctl error lines) with likely RHEL/AlmaLinux causes, instead of the misleading group hint. Test: setup-linux.bats daemon-won't-start case; bats 121 green. NOTE: this fixes the installer's HANDLING. Asad's root cause (why dockerd exits 1 on that box) is still masked by the systemd throttle and is being chased separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): load the kernel modules Docker/k3s need (fixes dockerd on minimal RHEL/AlmaLinux) Root cause of Asad's AlmaLinux 9 / EC2 failure: dockerd died on startup with "failed to register bridge driver: iptables ... addrtype ... missing kernel module". Minimal RHEL/AlmaLinux cloud images (incl. AWS) ship kernel-modules-core but NOT the full kernel-modules package, so xt_addrtype (+ br_netfilter, overlay) aren't available and Docker can't program its bridge NAT rules. New _ensure_kernel_modules() (setup-linux.sh), called before starting Docker: modprobe overlay / br_netfilter / xt_addrtype / iptable_nat / ip_tables; on RHEL-family, if a load fails, `dnf install kernel-modules-$(uname -r)` and retry; persist to /etc/modules-load.d for reboots. Best-effort + idempotent (verified clean on a healthy Ubuntu box). Also sharpened the daemon-won't-start hint to point at the kernel-modules remedy when the error mentions addrtype/missing module. This is the hospital-VM profile (minimal RHEL/Alma), so it's a real install-side fix, not just error handling. Test: setup-linux.bats _ensure_kernel_modules; bats 122. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(installer): cross-distro prereq matrix + real-Windows Pester + static gate Adds .github/workflows/installer-tests.yaml to validate the installer across the breadth of environments customers actually run — not just Ubuntu-amd64: • static — shellcheck (clean at --severity=warning) + bash -n + PSScriptAnalyzer • unit-bash — bats (mocked), 124 tests • unit-pester — Pester on Linux pwsh AND real windows-latest (the .ps1's true target) • distro-prereqs — NEW: runs the REAL Linux prereq path (PM detect, system deps, Docker branch, kernel modules, kubectl/k3d/helm) in a fresh container per distro family: ubuntu 22.04/24.04, debian 12, almalinux 9/8, rockylinux 9, amazonlinux 2023, fedora, opensuse leap. The matrix paid for itself before it even shipped: validating it locally against real distro containers surfaced a genuine gap — minimal Amazon Linux 2023 ships no openssl/tar, so helm's get-helm-3 fails ("openssl must first be installed"). Fixed in install_system_deps (ensure openssl + tar; package names are uniform across apt/dnf/yum/zypper/pacman), with bats coverage. All 9 validated distro branches now install every prerequisite. Installer-test jobs moved out of helm-ci.yaml into their own workflow (no more duplicate runs; helm-ci no longer triggers on scripts/** changes). Arch omitted (x86-only image + bare-container keyring friction; pacman branch covered by bats). Real k3d cluster-up (e2e) intentionally deferred — needs a stubbed backend; tracked. Validated locally via mac Docker: ubuntu:22.04, almalinux:9, amazonlinux:2023, opensuse/leap:15.6 → all PASS; bats 124 green; shellcheck 0 findings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(installer): fix static gate (.bats ≠ bash) + Windows-safe Confirm-Config test The new workflow's first run caught two issues — exactly its job: 1. Static analysis failed: `bash -n` ran on .bats files, which are bats DSL (@test "name" { … }), not valid bash. Restrict the syntax check to *.sh; .bats are validated by being run in the unit-bash job. 2. Pester on real windows-latest failed 1/55: the Confirm-Config test set $env:USERPROFILE = $env:HOME, but $env:HOME is empty on Windows, so [System.IO.Path]::GetFullPath("") threw "path is empty". The INSTALLER is correct (defaults to $env:USERPROFILE, always set on Windows) — the test fixture was Linux-centric. Derive a profile dir valid on both OSes. For the record, the first run's wins: all 9 distro prereq jobs (ubuntu 22.04/ 24.04, debian 12, almalinux 8/9, rockylinux 9, amazonlinux 2023, fedora, opensuse leap) + bats + Linux Pester passed on GHA's amd64 runners. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(installer): gate ShellCheck at error severity (SC2034 cross-file false positives) The libs are sourced together as one program, so single-file shellcheck reports SC2034 "unused" for shared vars defined in common.sh and consumed in other sourced files (CURL_SECURE, ARCH_DL, colours…). Gate at --severity=error (0 findings); warnings still printed as advisory. With this, Static analysis joins the already-green distro matrix + Windows/Linux Pester + bats. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(installer): real k3d cluster-up E2E on Ubuntu (amd64 + arm64) Adds scripts/tests/e2e-cluster.sh + an e2e-cluster matrix job — the highest- fidelity check CI can run. It drives the installer's OWN create_cluster() to bring up an actual k3d cluster on a real kernel (Docker is preinstalled on the runner), asserts every node reaches Ready, then proves the cluster can pull, schedule, and run a public workload (nginx:alpine), and tears down. It deliberately stops BEFORE the tracebloc helm install / backend registration (private images + real credentials), so it needs no secrets. Runs on ubuntu-22.04, ubuntu-24.04, and ubuntu-24.04-arm (arm64 runners are free on this public repo) — covering the real cluster path on both architectures. Validated locally on an arm64 Ubuntu VM: create_cluster() → server+agent Ready (k3s v1.29.4) → nginx pod Running → teardown. shellcheck clean (0 errors). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(installer): fix E2E probe race on the default ServiceAccount The amd64 runners proved the cluster comes up fine (nodes Ready), but the probe pod failed with "serviceaccount default not found" — kubectl run raced the SA controller, which creates default/default asynchronously after the node goes Ready. arm64 dodged it by timing. Wait for the SA before running the pod. Pure test-harness fix; the installer cluster path is correct on all arches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(installer): authenticated corporate-proxy E2E (squid) Adds scripts/tests/e2e-proxy.sh + an e2e-proxy job. Stands up a squid that REQUIRES basic auth, brings up a k3d cluster via the installer's create_cluster() with HTTP(S)_PROXY=http://user:pass@host.k3d.internal:3128, and proves the nodes pull a workload image THROUGH the authed proxy — the squid access log shows an authenticated CONNECT to auth.docker.io (which only a real image pull makes, never the readiness probe), closing the "proxy silently bypassed" false positive. It also asserts anonymous requests are refused, so auth is genuinely enforced. Guards the corporate-proxy hardening end-to-end (#172/#174, the Charité/hospital archetype): _write_k3d_proxy_config passes proxy env via a k3d config FILE so the '@' in user:pass@host survives (k3d splits --env on '@'), plus _augment_no_proxy. If the credentials regress, squid 407s and the pull hangs — the test fails loudly. Stops before the helm install / backend registration; no secrets. Validated locally on an arm64 Ubuntu VM: anonymous refused → cluster up via the authed proxy → nginx pulled through it (auth.docker.io + registry-1.docker.io CONNECTs by the proxy user) → teardown. shellcheck clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): install kernel-modules-extra + handle reboot-required (#176) dockerd crash-loops on minimal RHEL/AlmaLinux images because xt_addrtype/ iptable_nat/br_netfilter live in kernel-modules-extra, not the base kernel-modules package. The prior fix installed kernel-modules-$(uname -r) — the wrong package — so the self-heal never took. Install kernel-modules-extra (unversioned). When the repo's extra modules target a newer kernel than the running one (stale AMI), they can't load until reboot: detect that, set KMODS_REBOOT_REQUIRED, and have install_docker_engine print a clear reboot-and-re-run message instead of a raw Docker error. Modules persist via /etc/modules-load.d/tracebloc.conf. Verified end-to-end on a pristine AlmaLinux 10.1 MINIMAL EC2 box: reboot gate fires, post-reboot modules load, re-run reaches Connected. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Asad Iqbal <asad.dsoft@gmail.com> * chore(chart): bump version to 1.4.3 for installer-hardening release (#177) (#178) Installer-only patch release — promotes #171–#175 (RHEL-family support, credential/readiness verification, preflight gate, reboot persistence, --diagnose bundle) to production via develop→main. Chart templates/values are unchanged from 1.4.2. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: lukasWuttke <54042461+LukasWodka@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
LukasWodka added a commit
that referenced
this pull request
Jun 3, 2026
Users had no easy way to see which client version they're on (the CLI isn't shipped yet; `helm list` needs the namespace, and nothing surfaced it). Show the chart version where they already look: - install summary: a "Version" line next to Workspace. - --diagnose: as the first console line + recorded in the bundle header (the #1 thing support needs). Adds a best-effort `_chart_version` / Get-ChartVersion helper (greps helm's CHART column -> no jq). bash + PowerShell; bats + Pester coverage added. Verified: summary.bats + diagnose.bats green locally; ps1 via CI Pester. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
saadqbal pushed a commit
that referenced
this pull request
Jun 4, 2026
…d, surface version (#192) * fix(chart): drop the data-plane PriorityClass by default The cluster-scoped, fixed-name `tracebloc-data-plane` PriorityClass was the only thing forcing one tracebloc client per cluster (a second release collided on it with a cryptic Helm error) and blocking multiple tracebloc namespaces in one BYO cluster. mysql doesn't need it: memory requests==limits (last evicted under memory pressure), data on a PVC (eviction = transient restart, not data loss), and a PDB guards voluntary disruptions. Its only unique benefit was letting the scheduler preempt training jobs to keep mysql scheduled on a packed node — a narrow case. Default priorityClass.create=false + name="" so new installs template no PriorityClass and mysql carries no priorityClassName. Opt back in (create:true + name) on contended clusters, or reference an out-of-band one (create:false + name:<existing>). helm-unittest updated; 144/144 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): fixed namespace + one-per-machine guard; drop workspace prompt The "Choose a workspace name" prompt asked the user to invent a label that isn't their identity (the backend identifies a client by its credentials, not this string — it's just the local k8s namespace / Helm release name; the installer even discards the auth response body). It defaulted to a meaningless "default" and was the field that collided on a second install. - Drop the prompt; TB_NAMESPACE defaults to a fixed "tracebloc" (env-overridable for advanced/GitOps setups). - One-client-per-machine guard: after credentials verify, compare the entered Client ID against any client already installed here (helm get values). Same ID = a normal re-run/upgrade; a DIFFERENT ID hard-blocks with an explanation and options (repair / switch via `k3d cluster delete` / use another machine) instead of silently re-pointing the machine. This replaces the accidental PriorityClass collision (now dropped) with an intentional, explained guard. - Document the TB_NAMESPACE override; update bats (input sequences + 2 new guard tests). bats 26/27 — the 1 failure is a pre-existing macOS-bash-3.2 quirk in _extract_yaml_value, unrelated (CI bash 5 passes it). NOTE: install-k8s.ps1 + its Pester tests still need the same mirror (follow-up). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer-ps): mirror fixed namespace + one-per-machine guard (PowerShell) Mirrors the bash change in install-k8s.ps1: - drop the "Choose a workspace name" prompt; TB_NAMESPACE defaults to a fixed "tracebloc" (override via $env:TB_NAMESPACE). - one-client-per-machine guard: after credentials verify, compare the entered Client ID against any client already installed here (helm get values); a different ID hard-blocks with the same explanation/options as bash. - Pester: 2 new guard tests (block-different / allow-same). The existing Install-ClientHelm tests use dispatch-by-prompt Read-Host mocks, so the prompt removal doesn't disturb them. No pwsh locally -> verified via CI (Pester ubuntu+windows + PSScriptAnalyzer). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): scan all namespaces in the one-per-machine guard The guard checked only the `tracebloc` namespace, so a client installed by an older installer version (default namespace `default`, or a custom name) wasn't detected -- a re-run could create a second coexisting client. Now enumerate all client-chart releases (helm list -A) and compare each one's clientId, covering both fresh and migrated installs. bash uses jq (already a dependency; falls back to the tracebloc namespace if absent); PowerShell uses ConvertFrom-Json. The block message names the namespace. bats + Pester guard tests updated. Verified: bats green locally (jq path); PowerShell via CI Pester. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): show client (chart) version in summary + --diagnose Users had no easy way to see which client version they're on (the CLI isn't shipped yet; `helm list` needs the namespace, and nothing surfaced it). Show the chart version where they already look: - install summary: a "Version" line next to Workspace. - --diagnose: as the first console line + recorded in the bundle header (the #1 thing support needs). Adds a best-effort `_chart_version` / Get-ChartVersion helper (greps helm's CHART column -> no jq). bash + PowerShell; bats + Pester coverage added. Verified: summary.bats + diagnose.bats green locally; ps1 via CI Pester. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
saadqbal added a commit
that referenced
this pull request
Jun 4, 2026
* fix(resource-monitor): always grant read-only ClusterRole (decouple from clusterScope) Under clusterScope: false the chart rendered only a namespace-scoped Role in the release namespace. But the resource-monitor's code: * calls core_v1_api.list_pod_for_all_namespaces(field_selector=spec.nodeName=...) -- a CLUSTER-SCOPED list verb a namespaced Role can never satisfy; and * read_namespaced_pod()s its OWN pod, which lives in .Values.nodeAgents.namespace.name (NOT .Release.Namespace). So with clusterScope: false the DaemonSet 403'd on startup and crashlooped (70+ restarts observed on a live cluster). Per-node monitoring is intrinsically cluster-scoped. Always render the read-only ClusterRole + ClusterRoleBinding regardless of clusterScope (get/list/watch on pods/nodes/namespaces + metrics; no write, exec, or secret access). resourceMonitor: false still fully disables the component. clusterScope continues to gate the training/jobs isolation footprint elsewhere -- it must not leave the node monitor without permissions it cannot run without. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(resource-monitor): assert always-cluster-scoped RBAC under clusterScope=false Follow-up to the RBAC fix: node_agents_namespace_test.yaml still asserted the old behavior (namespaced Role + RoleBinding in the release namespace when clusterScope=false). Update that case to assert the corrected contract -- a ClusterRole + ClusterRoleBinding always render (with no metadata.namespace), while the subject SA still lives in the node-agents namespace. The clusterScope=false path stays under test; only the asserted behavior changes to match the fix. Verified with `helm unittest` (all resource-monitor suites pass). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(rbac): grant `get` on configmaps/secrets to jobs-manager SA The ingestion endpoint's orphan-resource verify path (client-runtime#52) and missing-row self-heal (client-runtime#54) read the existing ConfigMap/Secret on a create-409 to confirm content matches before reuse. The Role/ClusterRole only granted `create`, so those reads returned Forbidden and the endpoint 500'd instead of the intended 409/200-replay — verified live on the dev cluster. Add `get` alongside `create` in both the ClusterRole (clusterScope: true) and namespace Role (clusterScope: false) branches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(helm): guard that the pinned ingestor digest is multi-arch (closes#186) (#187) Add a helm-ci job (ingestor-multiarch) that parses images.ingestor.digest and fails the build unless it's a multi-arch index (linux/amd64 + linux/arm64). Greenfield installs spawn the ingestor Job from this PINNED digest before image-refresh first ticks, so an amd64-only pin breaks data ingestion on arm64 hosts (Apple Silicon, Graviton) with "no match for platform" / ImagePullBackOff. This would have caught #160 (the amd64-only v0.3.1 pin) before it shipped. ghcr.io/tracebloc/ingestor is public -> no secrets. Verified: passes on the current multi-arch baseline (sha256:d361fa77, v0.3.2 / #184), fails on the old amd64-only sha256:a0861ea9. Note: the digest is already multi-arch on develop as of v0.3.2 (#184 — the same d361fa77 index this PR previously bumped to), so #187 no longer touches values.yaml; it adds only the regression guard so an amd64-only pin can't slip back in. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(#190): fail image-refresh loudly when the ingestor (ghcr) digest can't resolve (#191) image-refresh silently skipped every tick when get_latest_digest returned empty for the ghcr.io ingestor image (egress/proxy/firewall to ghcr.io, or a blocked token endpoint) — never reaching the registry-drift branch that sets the new digest. jobs-manager + pods-monitor pull from docker.io and refreshed fine, so the CronJob looked healthy while the ingestor digest stayed pinned on the install-time baseline. That's why the berlin-team arm64 install sat on the amd64-only v0.3.1 digest even after :0.3 went multi-arch (#186 follow-up #2). Now count consecutive ingestor-resolve failures on a deployment annotation: - below imageRefresh.ingestorResolveFailureThreshold (default 3, ~45 min at the 15-min schedule) -> WARN + skip, as before (tolerate transient blips); - at/above it -> ERROR with actionable guidance, a tracebloc.io/ingestor-refresh-last-error annotation, and a non-zero exit so the Job fails visibly in `kubectl get cronjob` / monitoring — the same surfacing idiom Pass 2's stuck-rollout check already relies on; - a successful resolve clears the streak. Threshold is nil-guarded (default 3) for --reuse-values upgrades and schema-validated (integer >= 1). The digest-resolution logic itself is unchanged (verified correct: it returns the multi-arch index digest). helm unittest 146/146, helm lint clean, shellcheck + sh -n clean. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * test(requests-proxy): add helm-unittest coverage for requests-proxy Deployment (#194) requests-proxy-deployment.yaml was the only data-plane workload template without a unit test. This suite pins the properties most costly to regress: - security-context invariants (no SA-token automount, runAsNonRoot, seccomp RuntimeDefault, runAsUser 1001, no privilege escalation, drop ALL caps, read-only root filesystem) — see docs/SECURITY.md - the single-replica / single gunicorn worker constraint (the pod token registry is process-local; >1 worker silently shards token lookups) - the docker.io/tracebloc/jobs-manager image source and port 8888 - the nil-guarded resource defaults, plus an override case that exercises the default-through-dict fallthrough (guards the historic `readOnlyRootFilesystem: trueresources:` newline-eating regression) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(#196): allow training-pod egress to the requests-proxy (8888) (#197) The training-egress NetworkPolicy denies all pod-to-pod / ClusterIP egress (rule 2 excepts the cluster CIDRs) and re-permits only MySQL (rule 3). When the requests-proxy architecture shipped — training pods POST epoch results / FLOPs to requests-proxy-service:8888 instead of holding Service Bus credentials — this template was never updated to re-permit egress to the proxy. Result on every install with the policy enabled: pods hit "requests-proxy-service:8888 ... [Errno 111] Connection refused" at the first epoch finalize → CrashLoopBackOff → all experiments fail. Add rule 4 mirroring the MySQL rule: TCP/8888 to podSelector app=requests-proxy (same namespace). Service selector + port from templates/requests-proxy-service.yaml. Verified: `helm template -f ci/bm-values.yaml --show-only templates/network-policy-training.yaml` renders the new rule as valid YAML. Found live on a fresh client (tracebloc-amazon / k3d): jobs-manager reached the proxy (HTTP 401) while training pods got connection-refused — the only differentiator was this egress policy. Interim: live-patched the cluster + suspended its auto-upgrade CronJob (so reuse-values wouldn't revert the patch); re-enable once this lands + releases. Closes#196. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Installer UX: drop PriorityClass, fix namespace, one-per-machine guard, surface version (#192) * fix(chart): drop the data-plane PriorityClass by default The cluster-scoped, fixed-name `tracebloc-data-plane` PriorityClass was the only thing forcing one tracebloc client per cluster (a second release collided on it with a cryptic Helm error) and blocking multiple tracebloc namespaces in one BYO cluster. mysql doesn't need it: memory requests==limits (last evicted under memory pressure), data on a PVC (eviction = transient restart, not data loss), and a PDB guards voluntary disruptions. Its only unique benefit was letting the scheduler preempt training jobs to keep mysql scheduled on a packed node — a narrow case. Default priorityClass.create=false + name="" so new installs template no PriorityClass and mysql carries no priorityClassName. Opt back in (create:true + name) on contended clusters, or reference an out-of-band one (create:false + name:<existing>). helm-unittest updated; 144/144 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): fixed namespace + one-per-machine guard; drop workspace prompt The "Choose a workspace name" prompt asked the user to invent a label that isn't their identity (the backend identifies a client by its credentials, not this string — it's just the local k8s namespace / Helm release name; the installer even discards the auth response body). It defaulted to a meaningless "default" and was the field that collided on a second install. - Drop the prompt; TB_NAMESPACE defaults to a fixed "tracebloc" (env-overridable for advanced/GitOps setups). - One-client-per-machine guard: after credentials verify, compare the entered Client ID against any client already installed here (helm get values). Same ID = a normal re-run/upgrade; a DIFFERENT ID hard-blocks with an explanation and options (repair / switch via `k3d cluster delete` / use another machine) instead of silently re-pointing the machine. This replaces the accidental PriorityClass collision (now dropped) with an intentional, explained guard. - Document the TB_NAMESPACE override; update bats (input sequences + 2 new guard tests). bats 26/27 — the 1 failure is a pre-existing macOS-bash-3.2 quirk in _extract_yaml_value, unrelated (CI bash 5 passes it). NOTE: install-k8s.ps1 + its Pester tests still need the same mirror (follow-up). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer-ps): mirror fixed namespace + one-per-machine guard (PowerShell) Mirrors the bash change in install-k8s.ps1: - drop the "Choose a workspace name" prompt; TB_NAMESPACE defaults to a fixed "tracebloc" (override via $env:TB_NAMESPACE). - one-client-per-machine guard: after credentials verify, compare the entered Client ID against any client already installed here (helm get values); a different ID hard-blocks with the same explanation/options as bash. - Pester: 2 new guard tests (block-different / allow-same). The existing Install-ClientHelm tests use dispatch-by-prompt Read-Host mocks, so the prompt removal doesn't disturb them. No pwsh locally -> verified via CI (Pester ubuntu+windows + PSScriptAnalyzer). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): scan all namespaces in the one-per-machine guard The guard checked only the `tracebloc` namespace, so a client installed by an older installer version (default namespace `default`, or a custom name) wasn't detected -- a re-run could create a second coexisting client. Now enumerate all client-chart releases (helm list -A) and compare each one's clientId, covering both fresh and migrated installs. bash uses jq (already a dependency; falls back to the tracebloc namespace if absent); PowerShell uses ConvertFrom-Json. The block message names the namespace. bats + Pester guard tests updated. Verified: bats green locally (jq path); PowerShell via CI Pester. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): show client (chart) version in summary + --diagnose Users had no easy way to see which client version they're on (the CLI isn't shipped yet; `helm list` needs the namespace, and nothing surfaced it). Show the chart version where they already look: - install summary: a "Version" line next to Workspace. - --diagnose: as the first console line + recorded in the bundle header (the #1 thing support needs). Adds a best-effort `_chart_version` / Get-ChartVersion helper (greps helm's CHART column -> no jq). bash + PowerShell; bats + Pester coverage added. Verified: summary.bats + diagnose.bats green locally; ps1 via CI Pester. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * chore: bump chart 1.4.4 → 1.4.5 to ship the training-egress proxy fix (#198) 1.4.4 is already published on the tracebloc.github.io/client Pages channel and is what clusters run. The training-egress NetworkPolicy fix (#197, allow training → requests-proxy:8888) merged to develop without a version bump, so it is currently undeliverable: chart-releaser won't overwrite the existing 1.4.4 release, and clusters already on 1.4.4 would see no version change and pull nothing. Bump to 1.4.5 (lockstep version/appVersion, matching 1.4.3/1.4.4 history) so a v1.4.5 release publishes a new version that auto-upgrade actually pulls. Chart-only change; no image change. Ref #196 / #197. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: shujaat hasan <shujaathasan@shujaats-MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: shujaat_tracebloc <153823837+shujaatTracebloc@users.noreply.github.com> Co-authored-by: lukasWuttke <54042461+LukasWodka@users.noreply.github.com>
saadqbal added a commit
that referenced
this pull request
Jun 9, 2026
Review follow-up on #215 (comment #2). The miss-branch hint printed a bare `export PATH=…` (fixes only the current shell) followed by `source ${rc}` on an rc that did not yet contain the line — so neither command persisted the fix, while the closing note implied ${rc} should hold it. The user is never told to write the line into the rc. Rewrite the guidance per-shell: - POSIX shells (zsh / bash / sh / dash): `echo '<export>' >> ${rc}` then `source ${rc}` — one copy-pasteable step that fixes THIS terminal and every new one. - fish: `fish_add_path "…"` already persists (a universal var) AND applies to the running shell, so drop the misleading `source ~/.config/fish/config.fish`. Tests (install-cli.bats): the zsh miss-path now asserts the `echo … >> ~/.zshrc` form; the fish case asserts no POSIX `export` and no `source`. bats 8/8 pass, shellcheck --severity=error gate clean, bash -n clean. NOTE: review comment #1 (fish fresh-shell probe using `command -v`, which fish's `command` builtin lacks) is NOT addressed here — it needs verification on a real fish and is tracked separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
6 tasks
saadqbal added a commit
that referenced
this pull request
Jun 9, 2026
* feat(installer): self-verify CLI usability post-install with a shell-correct PATH fix (#738) Step 5 installed the tracebloc CLI and then told the user "open a new terminal so it's on your PATH" — without ever proving a fresh terminal would actually find it. That is exactly the cli#61 failure mode (binary lands in ~/.local/bin, which a brand-new shell doesn't have on PATH), left undetected until the customer hits it. The installer is the last place to catch it. After the install attempt, self-verify and report precisely: - Probe `command -v tracebloc` in BOTH a fresh login shell ("$SHELL" -lic) and a non-login shell ("$SHELL" -ic) — they read different startup files (~/.profile vs ~/.bashrc), and cli#61 was "works in my login shell, missing in a plain `bash` subshell". - If found: confirm via `tracebloc version` and print a VERIFIED verdict. The canonical `tracebloc dataset push ./data` next step stays in the summary's "What to do next" — not duplicated here. - If a fresh shell would NOT find it: print the EXACT shell-correct fix for the user's actual $SHELL (zsh→~/.zshrc, bash+linux→~/.bashrc, bash+darwin→~/.bash_profile, fish→fish_add_path + ~/.config/fish/config.fish, else ~/.profile), not a generic "open a new terminal". Stays NON-FATAL by design: the client is already connected by Step 5, so the verification always returns 0 and is hardened against the orchestrator's `set -e`. Mirrored in install-k8s.ps1 (RefreshPath is the faithful "fresh terminal" probe on Windows, since the CLI installer edits the user-scope registry PATH). Tests: extend install-cli.bats (verified-command success, actionable shell-correct PATH hint on miss, fish-specific fix, non-fatal under `set -e`) and mirror in install-k8s.Tests.ps1 (Test-TraceblocCli: verified verdict, actionable hint, non-fatal when RefreshPath throws). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): make the #738 Windows CLI-verify Pester-safe on Linux CI Two follow-ups so the Pester jobs go green (they were the only red checks on #215; Pester is green on develop, so this PR introduced both): - install-k8s.ps1: the new $TRACEBLOC_CLI_INSTALL_DIR ran Join-Path on $env:LOCALAPPDATA at top level. The Pester suite dot-sources this script, and on the Linux runner $env:LOCALAPPDATA is null — Join-Path throws on a null -Path, aborting BeforeAll and failing the whole container (0/65). Guard it; "" placeholder off Windows since the value is only used there. - install-k8s.Tests.ps1: add a `function tracebloc { }` stub so `Mock tracebloc` can bind. Pester v5 only mocks commands that already exist (cf. the existing kubectl/docker/helm/k3d stubs); without it the "fresh-shell success" test threw CommandNotFoundException — the lone windows-latest failure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): make the #738 PATH-fix guidance actually persist Review follow-up on #215 (comment #2). The miss-branch hint printed a bare `export PATH=…` (fixes only the current shell) followed by `source ${rc}` on an rc that did not yet contain the line — so neither command persisted the fix, while the closing note implied ${rc} should hold it. The user is never told to write the line into the rc. Rewrite the guidance per-shell: - POSIX shells (zsh / bash / sh / dash): `echo '<export>' >> ${rc}` then `source ${rc}` — one copy-pasteable step that fixes THIS terminal and every new one. - fish: `fish_add_path "…"` already persists (a universal var) AND applies to the running shell, so drop the misleading `source ~/.config/fish/config.fish`. Tests (install-cli.bats): the zsh miss-path now asserts the `echo … >> ~/.zshrc` form; the fish case asserts no POSIX `export` and no `source`. bats 8/8 pass, shellcheck --severity=error gate clean, bash -n clean. NOTE: review comment #1 (fish fresh-shell probe using `command -v`, which fish's `command` builtin lacks) is NOT addressed here — it needs verification on a real fish and is tracked separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Asad Iqbal <asad.dsoft@gmail.com>
This was referenced Jul 23, 2026
aptracebloc added a commit
that referenced
this pull request
Jul 29, 2026
Bugbot + @saadqbal + a self code-review on client#458, all in slice-2's code: - id -un everywhere (gate, _provision default, probe): $USER diverges from the rootless daemon's user under su/cron, which wedged detection/provisioning (#1). - Re-verify the uidmap helpers are usable (present AND setuid|cap_setuid) after install, and return non-zero + warn (NOT error/exit) so run_prepare_host stays best-effort while the installer sudo-path hard-fails via `|| error` (#2 + self-review). - _idmap_helper_ok (common.sh): accept the setuid bit OR a cap_setuid filecap, so Arch's `shadow`/pacman path isn't false-rejected (#3). - Hand-off + run_prepare_host fallback compute a non-overlapping start via _next_subid_start (honoring TB_SUBUID_FILE/TB_SUBGID_FILE), not hardcoded 100000 (#4 + self-review path-override). - Hand-off command names the researcher (TB_PREPARE_USER=) — bare prepare-host provisions nothing, so it would have looped back to the same hand-off (#5). - Capture `usermod --help` before grepping — pipefail-safe (#6). bats: id -un mocks, filecaps accept/reject, gate hand-off (names user + computed start), _provision re-verify best-effort, run_prepare_host best-effort. R8 regen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aptracebloc added a commit
that referenced
this pull request
Jul 29, 2026
Bugbot + @saadqbal + a self code-review on client#458, all in slice-2's code: - id -un everywhere (gate, _provision default, probe): $USER diverges from the rootless daemon's user under su/cron, which wedged detection/provisioning (#1). - Re-verify the uidmap helpers are usable (present AND setuid|cap_setuid) after install, and return non-zero + warn (NOT error/exit) so run_prepare_host stays best-effort while the installer sudo-path hard-fails via `|| error` (#2 + self-review). - _idmap_helper_ok (common.sh): accept the setuid bit OR a cap_setuid filecap, so Arch's `shadow`/pacman path isn't false-rejected (#3). - Hand-off + run_prepare_host fallback compute a non-overlapping start via _next_subid_start (honoring TB_SUBUID_FILE/TB_SUBGID_FILE), not hardcoded 100000 (#4 + self-review path-override). - Hand-off command names the researcher (TB_PREPARE_USER=) — bare prepare-host provisions nothing, so it would have looped back to the same hand-off (#5). - Capture `usermod --help` before grepping — pipefail-safe (#6). bats: id -un mocks, filecaps accept/reject, gate hand-off (names user + computed start), _provision re-verify best-effort, run_prepare_host best-effort. R8 regen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aptracebloc added a commit
that referenced
this pull request
Jul 29, 2026
Bugbot + @saadqbal + a self code-review on client#458, all in slice-2's code: - id -un everywhere (gate, _provision default, probe): $USER diverges from the rootless daemon's user under su/cron, which wedged detection/provisioning (#1). - Re-verify the uidmap helpers are usable (present AND setuid|cap_setuid) after install, and return non-zero + warn (NOT error/exit) so run_prepare_host stays best-effort while the installer sudo-path hard-fails via `|| error` (#2 + self-review). - _idmap_helper_ok (common.sh): accept the setuid bit OR a cap_setuid filecap, so Arch's `shadow`/pacman path isn't false-rejected (#3). - Hand-off + run_prepare_host fallback compute a non-overlapping start via _next_subid_start (honoring TB_SUBUID_FILE/TB_SUBGID_FILE), not hardcoded 100000 (#4 + self-review path-override). - Hand-off command names the researcher (TB_PREPARE_USER=) — bare prepare-host provisions nothing, so it would have looped back to the same hand-off (#5). - Capture `usermod --help` before grepping — pipefail-safe (#6). bats: id -un mocks, filecaps accept/reject, gate hand-off (names user + computed start), _provision re-verify best-effort, run_prepare_host best-effort. R8 regen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aptracebloc added a commit
that referenced
this pull request
Jul 29, 2026
…#1220) (#458) * feat(install): Tier-1 subuid/subgid gate + prepare-host remediation RFC 0001 #1220. Detect the one privileged residue a modern rootless host may still need — a subordinate UID/GID range + the setuid uidmap helpers — and either proceed (present), hand off to prepare-host (unprivileged), or perform one announced touch (sudo available). Never blanket sudo, never an opaque mid-install crash inside dockerd-rootless-setuptool.sh. - probe.sh: _probe_subid_ranges (PROBE_SUBID) + _probe_uidmap_helpers (PROBE_UIDMAP), set in run_host_probes (Linux only), plus audit rows on the Tier-1 path. - common.sh: shared pure parsers _subid_has_entry + _next_subid_start, used by both the probe and the remediation (no duplication). - setup-linux.sh: _ensure_subid_ranges gate (present / hand-off / one announced sudo touch) called before install_rootless_docker; _provision_subid_ranges (idempotent, non-overlapping block, usermod --add-subuids with file-append fallback, uidmap install) shared by the installer and run_prepare_host. Folds in slice-1's minimal uidmap check. - Tests: probe.bats + setup-linux.bats. Manifest regenerated (R8). Closestracebloc/backend#1220 Part of tracebloc/backend#1177 · Epic tracebloc/backend#1168 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): address #458 review — subid gate/probe/hand-off robustness Bugbot + @saadqbal + a self code-review on client#458, all in slice-2's code: - id -un everywhere (gate, _provision default, probe): $USER diverges from the rootless daemon's user under su/cron, which wedged detection/provisioning (#1). - Re-verify the uidmap helpers are usable (present AND setuid|cap_setuid) after install, and return non-zero + warn (NOT error/exit) so run_prepare_host stays best-effort while the installer sudo-path hard-fails via `|| error` (#2 + self-review). - _idmap_helper_ok (common.sh): accept the setuid bit OR a cap_setuid filecap, so Arch's `shadow`/pacman path isn't false-rejected (#3). - Hand-off + run_prepare_host fallback compute a non-overlapping start via _next_subid_start (honoring TB_SUBUID_FILE/TB_SUBGID_FILE), not hardcoded 100000 (#4 + self-review path-override). - Hand-off command names the researcher (TB_PREPARE_USER=) — bare prepare-host provisions nothing, so it would have looped back to the same hand-off (#5). - Capture `usermod --help` before grepping — pipefail-safe (#6). bats: id -un mocks, filecaps accept/reject, gate hand-off (names user + computed start), _provision re-verify best-effort, run_prepare_host best-effort. R8 regen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(install): stub the subid gate in the Tier-1 rootless routing test install_linux's Tier-1 branch now calls _ensure_subid_ranges (slice 2) before install_rootless_docker; the routing test left it un-stubbed, so the real gate hit the no-sudo hand-off and error()'d → install_linux returned non-zero. Stub _ensure_subid_ranges (its own behavior is covered by the dedicated gate tests) and assert it runs before daemon setup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(install): scope set -o pipefail to a subshell (bats harness footgun) Setting `set -o pipefail` in the @test body can leak into bats' own post-test pipelines and fail the whole run with exit 1 even when every test reports ok (no 'not ok'). Confine it to a subshell around the call so the pipefail-safety assertion still holds without touching the harness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): style guard — no bare curl in the prepare-host hint The hand-off piped 'curl … | TB_PREPARE_USER=… bash', which breaks check-style.sh's exemption for the canonical 'curl … | bash' one-liner (the env var sits between the pipe and bash). Split into an 'export TB_PREPARE_USER=…' line + the canonical piped one-liner — still names the researcher, and passes the guard. Verified with scripts/check-style.sh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): fix the #458 red bats + 2 Bugbot bugs (newgidmap cap, write-failure) Root cause of the "540 ok but exit 1" bats red: the probe.bats uidmap tests set PATH="$bin" in the test body to hide system helpers, which also hides `rm` — so bats-core 1.10+ can't run its own per-test cleanup ("rm: command not found") and fails the whole run even though every test passes. Scope the hermetic PATH to a subshell so it can't leak into bats' machinery. (Why develop was green + this was so hard to see: these tests are new in slice 2, and the symptom is a clean pass list with a non-zero exit.) Two real Bugbot findings in the slice's own code: - _idmap_helper_ok checked cap_setuid for BOTH helpers; newgidmap carries cap_setgid (Arch filecaps) -> false-rejected. Map name->cap; fix the test mock that masked it + add a wrong-cap regression test. - _provision_subid_ranges printed success/returned 0 even when the usermod/tee write failed (callers run it with set -e off) -> installer proceeds with no range. Guard every write; warn + return 1 on failure. + a test. Verified: probe.bats + setup-linux.bats EXIT 0 (0 not-ok, 0 rm-not-found) in a faithful ubuntu 24.04 + bats 1.10 + non-root container. Rebased onto develop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): 2 more Bugbot findings on #458 (apt hang, false zero-root message) - _install_uidmap_pkg ran a bare `sudo apt-get install -y uidmap` under the spinner — no needrestart/DEBIAN_FRONTEND env, no DPkg::Lock::Timeout, no apt_wait_for_lock — so a headless Tier-1 install can hang on Ubuntu needrestart or an apt-daily lock (#210 class). Reuse the repo's hardened PM_INSTALL (populate via setup_pm, which Tier 1 skips) + apt_wait_for_lock. - install_rootless_docker always printed "no administrator rights were used", even after _ensure_subid_ranges performed an announced sudo touch on the root/sudo_nopw path. The gate now sets TB_ROOTLESS_ADMIN_TOUCH and the summary is honest on both the zero-root and one-admin-touch paths. Tests: hardened-install assertion (NEEDRESTART_MODE + DPkg::Lock::Timeout) + a success-message honesty test. Verified EXIT 0 (0 not-ok, 0 rm-errors) in the faithful ubuntu 24.04 + bats 1.10 + non-root container. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): refresh the package index in _install_uidmap_pkg (Bugbot #458) Completing the prior apt-hardening: _install_uidmap_pkg populated PM_INSTALL and waited for the dpkg lock but never ran PM_UPDATE. On the Tier-1 path this is the first package op, so an empty/stale index can't locate uidmap/shadow and the install hard-stops. Run $PM_UPDATE (best-effort) first, matching the repo's other install paths (setup-linux.sh:335/543). Test asserts the index refresh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aptracebloc added a commit
that referenced
this pull request
Jul 29, 2026
- _persist_docker_host: key idempotency off our own marker, not a bare 'DOCKER_HOST=' probe. The old probe also matched a user's own DOCKER_HOST (remote/TCP), so we silently skipped persisting the rootless socket and new shells kept hitting the wrong daemon. Now: our own line -> idempotent; a foreign DOCKER_HOST -> left untouched + a warn to repoint it (Asad #2 + Bugbot #478, Medium). - ensure_cluster_autostart: reset TB_DOCKER_AUTOSTART=0 in the rootless else-branch (defensive; the is-enabled seed is already guarded off the rootless path) so the honesty guarantee is local to the branch (Asad #1). - install_rootless_docker: success line now reads "one or more one-time admin steps" so it doesn't undercount when both the subuid and cgroup touches happen (Saqlain #1). - Test: foreign DOCKER_HOST -> warns, no clobber, no double-write. manifest regen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aptracebloc added a commit
that referenced
this pull request
Jul 29, 2026
… (#1221) (#478) * feat(install): Tier-1 k3d-on-rootless + cgroup delegation + autostart (#1221) Slice #1221 (RFC 0001 / #1177): make a rootless Tier-1 cluster actually usable, all behind the opt-in TB_TIER1_ROOTLESS flag (default off until the spike's §5 host validation). With the flag unset every path below is a no-op and current behavior is byte-for-byte unchanged. - Shared _rootless_active predicate (common.sh) so cluster.sh + setup-linux.sh can't drift on the flag pair. - create_cluster targets the rootless socket (DOCKER_HOST); ensure_cluster_ autostart gets a user-scope branch (systemctl --user enable + loginctl enable-linger, never `sudo systemctl enable docker`), and promises reboot-survival only when BOTH succeed (honesty rule, #375/#458). - cgroup v2 controller delegation drop-in (Delegate=cpu cpuset io memory pids): privileged write + daemon-reload on root/sudo, or hand off to prepare-host with the exact path+content when unprivileged. run_prepare_host writes it too (system-wide -> covers the researcher). - Carry-ins from #452/#458: scope-aware _configure_docker_proxy (user scope, no sudo) so a proxy-only host's rootless daemon can pull rancher/k3s; _set_tools_target installs user-space on rootless Tier 1 (no sudo-mv crash on a true no-sudo host); persist DOCKER_HOST to the shell rc for new terminals. 14 new bats tests incl. flag-off regressions; shellcheck --severity=error clean; manifest.sha256 regenerated (R8). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): address Bugbot on #478 — rootless autostart seed + admin-touch msg - ensure_cluster_autostart: don't seed TB_DOCKER_AUTOSTART from the SYSTEM docker.service is-enabled check on the rootless path. The cluster runs on the per-user rootless socket, so a system unit that happens to be enabled would seed a false reboot promise the rootless branch then can't honestly retract. On rootless the user-scope enable+linger are now the sole authority (Bugbot medium). - install_rootless_docker: the TB_ROOTLESS_ADMIN_TOUCH success line no longer hardcodes "subuid/subgid range" — _ensure_cgroup_delegation can set that flag too, so it now names "host prerequisites (subuid/subgid range and/or cgroup delegation)" (Bugbot low). - Test: rootless + system docker.service enabled + user-enable fails => flag stays 0 (pins the seed-guard). manifest regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): address Asad + Saqlain review on #478 - _persist_docker_host: key idempotency off our own marker, not a bare 'DOCKER_HOST=' probe. The old probe also matched a user's own DOCKER_HOST (remote/TCP), so we silently skipped persisting the rootless socket and new shells kept hitting the wrong daemon. Now: our own line -> idempotent; a foreign DOCKER_HOST -> left untouched + a warn to repoint it (Asad #2 + Bugbot #478, Medium). - ensure_cluster_autostart: reset TB_DOCKER_AUTOSTART=0 in the rootless else-branch (defensive; the is-enabled seed is already guarded off the rootless path) so the honesty guarantee is local to the branch (Asad #1). - install_rootless_docker: success line now reads "one or more one-time admin steps" so it doesn't undercount when both the subuid and cgroup touches happen (Saqlain #1). - Test: foreign DOCKER_HOST -> warns, no clobber, no double-write. manifest regen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LukasWodka added a commit
that referenced
this pull request
Jul 30, 2026
* chore(chart): close values-schema gaps + drop dead override/code/docs (#963) (#457) * chore(chart): close values-schema gaps + drop dead override/code/docs (#963) Contract fixes for the client Helm chart (re-verified against develop at chart v1.9.6; the #963 audit was taken at v1.8.4): - values.schema.json: add the six live-but-unvalidated keys so bad values fail `helm lint` instead of silently passing — egressReachabilityCheck.enabled, ingestionAuthz.{allowed,serviceAccountName}, networkPolicy.training.enforcementProbeTimeoutSeconds, podTokenSigningSecret, podTokenTtlSeconds. Types/defaults/constraints taken from values.yaml and the templates that consume them. helm lint passes. - ingestor subchart: remove the dead `image.repository` key — no template ever rendered it (jobs-manager spawns from the parent chart's images.ingestor.repository). Kept image.digest (live). README's air-gapped override rows now point at the authoritative parent-chart path. - README: drop the hardcoded chart version (said v1.3.5 while Chart.yaml is 1.9.6) and point to Chart.yaml / the releases page, so it can't drift again. - Delete the unwired check_docker_arch_mac function + its bats test (no call sites) and the orphaned docs/eks.md (referenced nowhere). Part of tracebloc/backend#963. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(installer): regenerate manifest after common.sh trim + develop merge The #963 chart-contract cleanup dropped 48 dead lines from scripts/lib/common.sh, changing its sha; the installer manifest wasn't regenerated, so the Static analysis gate (gen-manifest.sh --check) failed. Merging develop also refreshed preflight.sh/install-k8s.ps1 hashes. Regenerate scripts/manifest.sha256 to match the working tree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(release): package charts as the tag version + pre-releases skip gh-pages (#467) * fix(release): package charts AS the tag version + pre-releases skip gh-pages Incident 2026-07-29: the v1.9.7-rc.1 pre-release packaged the client chart from Chart.yaml's plain 1.9.7 and pushed it into the public helm index as a STABLE version -- customers running helm upgrade would have received staging content (removed from the index by hand, tgz deleted). Two layers now prevent it: (1) helm package --version/--app-version from the release tag, so rc charts carry the -rc.N suffix helm's pre-release rules key on; (2) pre-releases never run the gh-pages index steps at all -- FR consumes the release assets (stamped installer / chart tgz), the index is a customer surface reserved for finals. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gh-pages gates key on verify's tag-derived prerelease, not the frozen event (Bugbot) github.event.release.prerelease is an event-time snapshot: after verify demotes a mis-marked release, it still reads false, so the demoted rc would have entered the public index anyway. verify now outputs effective prerelease-ness derived from the tag shape (the same strict rule the demotion uses) and all three gh-pages steps gate on that output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: post-publish index-invariant job (manual leak catch -> CI) After every release run: the public index must contain only stable-shaped versions, and a prerelease run must not have indexed its own version. Fails loudly; would have caught the 1.9.7 leak within a minute of it happening instead of during manual FR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(install): rootless Docker core + Tier-1 routing (opt-in) (#1219) (#452) * feat(install): rootless Docker core + Tier-1 routing (opt-in) Add install_rootless_docker() and a Tier-1 early-branch in install_linux so a modern-kernel host with no runtime and no root can install entirely in user space (RFC 0001 Tier 1 — the RFC's primary path). Gated behind opt-in TB_TIER1_ROOTLESS=1; with the flag unset a Tier-1 host falls through to the legacy privileged flow unchanged (validated default). - install_rootless_docker: uidmap-helper precondition (defers to prepare-host #1178 when absent — never self-sudo), no-sudo install via dockerd-rootless-setuptool.sh or get.docker.com/rootless, user-scoped systemctl --user + loginctl enable-linger, DOCKER_HOST export with XDG_RUNTIME_DIR fallback, single docker-info verify (no retry loop). - Tier-1 branch mirrors the Tier-0 early-return. Tools still install via sudo here (_set_tools_target keys no-sudo off Tier 0 only) — tightening that for rootless Tier 1 is deferred to slice 3 (#1221). - 6 bats cases; scripts/manifest.sha256 regenerated (R8). Closestracebloc/backend#1219 Part of tracebloc/backend#1177 · Epic tracebloc/backend#1168 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address Bugbot review on rootless Docker core (#452) - Prepend ~/bin to PATH after the rootless install so this run's docker info verify + later k3d/docker calls resolve the CLI the get.docker.com/rootless fallback installs there (High). - Bound the rootless `docker info` verify with a new shared _bounded helper (timeout/gtimeout, mirrors probe.sh) so a wedged user daemon can't hang a headless install (Medium). - Guard the user-systemd bring-up under set -e: `systemctl --user … || true` (the bounded verify is the real gate) and `loginctl enable-linger … || warn` (optional; fails on polkit-locked hosts even when the daemon is up) (Medium). Adds 2 bats cases (~/bin on PATH; systemd/linger failure falls through to the verify). Manifest regenerated (R8). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: correct the uidmap remedy message (Bugbot #452) The missing-uidmap error claimed prepare-host would install the uidmap helpers, but run_prepare_host only sets up privileged Docker + the docker group — it never installs uidmap. Point at the two honest remedies instead: install the `uidmap` package directly (rootless then works), or run prepare-host to set up Docker so the researcher installs at Tier 0 (no rootless needed). #1220 folds this into the shared subuid/subgid gate and teaches prepare-host to install uidmap for real. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(install): TODO(#1221) — rootless daemon needs user-scoped proxy config Bugbot on #452 flagged that install_rootless_docker never configures a corporate proxy for the user-scoped dockerd (the #244 _configure_docker_proxy is sudo/system-scoped and the Tier-1 early-return never reaches it), so k3d pulls of rancher/k3s time out on proxy-only hosts. Deferred to #1221 (the k3d-on-rootless-socket slice that owns the pulls); leaving a tracked TODO so the follow-up adds the user-scoped drop-in. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address Asad review nits on #452 — chmod no-op + misleading comment - Drop the chmod +x on the rootless installer script: it runs via `sh "$rootless_script"`, which ignores the exec bit. - Reword the Tier-1 _install_userspace_tools comment: tools still sudo-install on Tier 1 (only _persist_tools_on_path is no-sudo); the comment previously implied otherwise. The underlying _set_tools_target sudo-crash on no-sudo hosts and the post-install DOCKER_HOST shell persistence are tracked to #1221. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: resolve user via id -un in install_rootless_docker (Saqlain review, #452) $USER can be empty in headless / su / cron contexts (a Tier-1 target), which would break `loginctl enable-linger` and the success line. Resolve the user once via `id -un` (fallback $USER) and use it for the linger call, its hint, and the success message. Matches the id-based robustness DOCKER_HOST already uses. Happy-path bats now mocks `id -un` cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: add code-quality caller workflow (advisory) (#463) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * Add pre-commit hooks (Layer 0, lint-only) (#465) * Add pre-commit config (Layer 0, lint-only) Lint-only on purpose: scripts/manifest.sha256 must keep matching the bytes under scripts/, so no hook may rewrite files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Document pre-commit setup in README Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * docs: add Bugbot resolve-and-reply team norm to .cursor/BUGBOT.md (#464) Part of tracebloc/backend#1308 Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * ci: cover scripts/resolve-ingestor-digest.sh in CI shellcheck (#466) * ci: lint scripts/resolve-ingestor-digest.sh in CI shellcheck (was never linted) Both CI shellcheck invocations enumerate files explicitly and both omitted this script. Verified clean against shellcheck --severity=error --shell=bash 0.11.0 before adding. The pre-commit hook from #465 already covers it locally; this closes the same gap on the CI side. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: lint scripts/resolve-ingestor-digest.sh in CI shellcheck (was never linted) Both CI shellcheck invocations enumerate files explicitly and both omitted this script. Verified clean against shellcheck --severity=error --shell=bash 0.11.0 before adding. The pre-commit hook from #465 already covers it locally; this closes the same gap on the CI side. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): honest cosign bootstrap download + translated DISM failures (#468) (#469) The v1.9.7-rc.1 FR killed a healthy install: PS 5.1's progress overlay throttled the 17 MB pinned-cosign fetch to ~4.5 min of dead silence and the window read as frozen. - silence the PS 5.1 progress overlay in Get-WithRetry/Get-Optional (function-local, auto-reverts) - the classic 10-50x IWR speedup - run the cosign fetch in a background job with a dim liveness tick (Wait-JobWithTicks / Get-OptionalWithTicks; cwd pinned per #409, TLS 1.2 re-applied in the fresh process), expectation lines before, elapsed + checksum-verified confirmation after - ASCII-only string literals in both installers: the release asset is served without a charset so PS 5.1's irm decodes UTF-8 source as Latin-1 before iex, and BOM-less -File reads are ANSI - literal em-dashes/ellipses reached customers as mojibake. Locked in by a tokenizer-based Pester test (which also caught the -Help here-string). - Enable-OneVirtFeature: translate DISM's raw COMException (feature package absent on Server SKUs vs enable failure) and stop demanding a reboot for a feature that never enabled (old code sent Server users into a reboot->re-run->same-error loop) Pester: 212 passed / 0 failed locally (pwsh 7.5, Pester 5.7.1). PSScriptAnalyzer: 0 errors. manifest.sha256 regenerated. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * fix(installer): trust the corporate MITM CA in the k3d nodes (#424) (#453) * fix(installer): trust the corporate MITM CA in the k3d nodes (#424) Proxy REACHABILITY reaches the nodes, but on a TLS-inspecting (break-and- inspect) network the nodes still don't TRUST the corporate CA, so every in-node containerd pull (rancher/k3s, ghcr.io, tracebloc images) fails x509 — then masked (helm runs without --wait) into a root-cause-free "an image couldn't be pulled." Enterprise/hospital archetype, all three OSes. - Inject the CA at create time: when TRACEBLOC_CA_BUNDLE (or CURL_CA_BUNDLE) is set, mount the bundle into every k3d node and write a registries.yaml pointing containerd at it per-registry (docker.io, registry-1.docker.io, ghcr.io), via the same --config/create path that already carries proxy env. Parity across scripts/lib/cluster.sh (Linux/macOS) and install-k8s.ps1 (Windows). A CA var set but unreadable fails loudly instead of silently skipping. - Name the env var where the user hits the wall: the TLS-interception preflight hint (both OSes), docs/INSTALL.md, and the PS -Help env-var list. - CA-aware diagnosis: detect x509 / "certificate signed by unknown authority" pull events and report a dedicated image_pull_ca state — "the cluster does not trust your network's TLS-inspection CA" + the exact remedy — instead of the generic pull error. Mirrored in summary.sh and Print-Summary. - New check-drift.sh parity check (_drift_ca_trust) so neither installer can drop the CA wiring for the other's OS. Tests: +8 cluster.bats, +3 summary.bats, +2 check-drift.bats, +8 Pester. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): CA-trust hardening — no fail-open, bounded events, verify CA readable (Bugbot #424) Three Bugbot findings on #424: - _write_k3d_registries_config failed open: on mktemp failure it returned success with no path, so create still mounted the CA and logged "nodes trust it" but dropped --registry-config → containerd never got ca_file, x509 pulls still fail while the operator thinks it's fixed. Now returns non-zero; the caller hard-errors (CA was supplied, so we refuse to proceed without wiring it in). - PS Get-NotReadyState `kubectl get events` had no --request-timeout (the bash path does) — on a wedged/proxy-misrouted API, classification could hang. Added --request-timeout=5s to match _diagnose_not_ready. - PS Resolve-CaBundle only checked existence (Test-Path), not readability, so an unreadable CA passed on Windows but bash (-r) hard-fails. Added an OpenRead probe so both fail the same way, up front. Tests: cluster.bats +mktemp-failure + unwritable-registries-hard-error; install-k8s.Tests.ps1 +unreadable-CA (Unix) + events --request-timeout assertion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): errexit-safe CA-resolve capture + drift check ignores comments (Bugbot #424 r2) Two round-2 Bugbot findings: - Under `set -euo pipefail`, `ca_bundle="$(_resolve_ca_bundle)"; ca_rc=$?` exited on the rc-2 (unreadable/missing CA) BEFORE ca_rc/error ran — operators got a bare exit instead of the "can't be read" guidance. Capture with `|| ca_rc=$?` so errexit doesn't fire and the guidance prints. - _drift_ca_trust whole-file grep matched tokens in comments (e.g. --registry-config appears in a comment above the real line), so deleting the functional wiring could still pass. Strip comment lines first (matches the execute-gate / preflight-host checks), no grep -q under pipefail. Tests: cluster.bats +errexit-safe-capture; check-drift.bats +comment-only-token drift. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): TLS-preflight hint names the right var per layer/OS (Bugbot #424 r3) The hint claimed TRACEBLOC_CA_BUNDLE makes "the host AND the k3d nodes" trust the CA, but the host connectivity checks use curl_secure / Invoke-WebRequest, which read CURL_CA_BUNDLE / the system trust store — not TRACEBLOC_CA_BUNDLE (that var only reaches the nodes via _resolve_ca_bundle). Following the hint literally left host preflight TLS failures unchanged. Corrected, no behaviour change: - bash: CURL_CA_BUNDLE fixes these host checks AND the nodes; TRACEBLOC_CA_BUNDLE is nodes-only; or add the CA to the system trust store. - Windows: import the CA into the cert store for the host checks (Invoke-WebRequest uses the store, not an env var); TRACEBLOC_CA_BUNDLE/CURL_CA_BUNDLE cover the nodes. (Reworded to avoid a bare lowercase `curl` that the curl_secure style guard flags.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): apply CA on cluster REUSE path — warn + recreate guidance (Bugbot #424 r4) The image_pull_ca remedy said "set the CA and re-run", but CA trust is baked in only at fresh create; a re-run reuses the existing cluster and never mounts the CA or passes --registry-config, so the x509 pulls persisted. Mirror the existing proxy handling (baked-at-create → warn on reuse): - bash _check_existing_cluster_ca (called from _handle_existing_cluster): warns when a CA bundle is set but the reused server container lacks the CA mount. - ps1 New-K3dCluster reuse block: same check via docker inspect mounts. - both image_pull_ca remedies now say to `k3d cluster delete <name>` first, then re-run with the CA (CA, like proxy, can't be added to a running cluster). Tests: cluster.bats +3 (no-CA no-op / CA-but-missing-mount warns / mount-present silent). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): add auth.docker.io to the CA registries config (Bugbot #424 r5) The registries.yaml ca_file entries covered docker.io / registry-1.docker.io / ghcr.io, but Docker Hub pulls also TLS-handshake with auth.docker.io for bearer tokens — so on a break-and-inspect network containerd still rejected the intercepted cert there even with the CA mounted. #416 already probes auth.docker.io at preflight; the CA registries list now matches. Added to TB_CA_REGISTRIES and $TbCaRegistries; registries.yaml test counts 3 -> 4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#424): _resolve_ca_bundle rejects a directory, not just unreadable paths A directory of PEMs is readable (-r) but would bind-mount over the single node ca_file path and containerd can't read it — the silent 'looks applied but still x509' case. Require a regular file (-f), mirroring the PS Resolve-CaBundle -PathType Leaf check. Adds a directory-reject bats case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#424): exact whole-line match for CA mount detection (Bugbot) _check_existing_cluster_ca used a substring test on docker mount destinations, so a longer path embedding /etc/ssl/certs/tracebloc-mitm-ca.crt (e.g. …crt.bak) would be treated as the CA mount and skip the recreate warning while containerd still x509-fails. Switch to grep -qxF (exact whole-line), matching the PS anchored regex. Adds a substring-embed test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#424): scope x509 classification to the pull-failure event (Asad) _diagnose_not_ready / Get-NotReadyState flagged image_pull_ca on ANY x509 event in the namespace, so a stale/unrelated x509 event (e.g. a FailedMount) could misdirect the user into a needless delete+recreate. Filter events to the image-pull failure lines (failed to pull / ErrImagePull) before testing x509, in both bash and PS. Adds an unrelated-x509 test to each side. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(chart): perIngestionTables — RFC-0003 D16 enablement knob (backend#1205) (#472) * feat(chart): perIngestionTables — the RFC-0003 D16 enablement knob (backend#1204/#1205) values.perIngestionTables (default false, schema-typed) renders PER_INGESTION_TABLES=1 onto the jobs-manager, which forwards it into every ingestion Job it spawns (client-runtime companion PR). Flip per environment, dev first, only once that environment's backend + engine images + jobs-manager carry the merged D-series. Default installs render byte-identically (conditional block; unit tests pin both sides). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(values): own banner for perIngestionTables — it is not part of the authz section (review) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * chore: clear house-rules findings (#470) Fix every finding the shared org checker (tracebloc/.github scripts/house-rules.sh) reports at develop HEAD: missing curl timeouts/TLS floors, plus (cli) a missing pipefail. Waivers only where the finding is a documented false positive. Part of tracebloc/backend#1303. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * feat(install): Tier-1 subuid/subgid gate + prepare-host remediation (#1220) (#458) * feat(install): Tier-1 subuid/subgid gate + prepare-host remediation RFC 0001 #1220. Detect the one privileged residue a modern rootless host may still need — a subordinate UID/GID range + the setuid uidmap helpers — and either proceed (present), hand off to prepare-host (unprivileged), or perform one announced touch (sudo available). Never blanket sudo, never an opaque mid-install crash inside dockerd-rootless-setuptool.sh. - probe.sh: _probe_subid_ranges (PROBE_SUBID) + _probe_uidmap_helpers (PROBE_UIDMAP), set in run_host_probes (Linux only), plus audit rows on the Tier-1 path. - common.sh: shared pure parsers _subid_has_entry + _next_subid_start, used by both the probe and the remediation (no duplication). - setup-linux.sh: _ensure_subid_ranges gate (present / hand-off / one announced sudo touch) called before install_rootless_docker; _provision_subid_ranges (idempotent, non-overlapping block, usermod --add-subuids with file-append fallback, uidmap install) shared by the installer and run_prepare_host. Folds in slice-1's minimal uidmap check. - Tests: probe.bats + setup-linux.bats. Manifest regenerated (R8). Closestracebloc/backend#1220 Part of tracebloc/backend#1177 · Epic tracebloc/backend#1168 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): address #458 review — subid gate/probe/hand-off robustness Bugbot + @saadqbal + a self code-review on client#458, all in slice-2's code: - id -un everywhere (gate, _provision default, probe): $USER diverges from the rootless daemon's user under su/cron, which wedged detection/provisioning (#1). - Re-verify the uidmap helpers are usable (present AND setuid|cap_setuid) after install, and return non-zero + warn (NOT error/exit) so run_prepare_host stays best-effort while the installer sudo-path hard-fails via `|| error` (#2 + self-review). - _idmap_helper_ok (common.sh): accept the setuid bit OR a cap_setuid filecap, so Arch's `shadow`/pacman path isn't false-rejected (#3). - Hand-off + run_prepare_host fallback compute a non-overlapping start via _next_subid_start (honoring TB_SUBUID_FILE/TB_SUBGID_FILE), not hardcoded 100000 (#4 + self-review path-override). - Hand-off command names the researcher (TB_PREPARE_USER=) — bare prepare-host provisions nothing, so it would have looped back to the same hand-off (#5). - Capture `usermod --help` before grepping — pipefail-safe (#6). bats: id -un mocks, filecaps accept/reject, gate hand-off (names user + computed start), _provision re-verify best-effort, run_prepare_host best-effort. R8 regen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(install): stub the subid gate in the Tier-1 rootless routing test install_linux's Tier-1 branch now calls _ensure_subid_ranges (slice 2) before install_rootless_docker; the routing test left it un-stubbed, so the real gate hit the no-sudo hand-off and error()'d → install_linux returned non-zero. Stub _ensure_subid_ranges (its own behavior is covered by the dedicated gate tests) and assert it runs before daemon setup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(install): scope set -o pipefail to a subshell (bats harness footgun) Setting `set -o pipefail` in the @test body can leak into bats' own post-test pipelines and fail the whole run with exit 1 even when every test reports ok (no 'not ok'). Confine it to a subshell around the call so the pipefail-safety assertion still holds without touching the harness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): style guard — no bare curl in the prepare-host hint The hand-off piped 'curl … | TB_PREPARE_USER=… bash', which breaks check-style.sh's exemption for the canonical 'curl … | bash' one-liner (the env var sits between the pipe and bash). Split into an 'export TB_PREPARE_USER=…' line + the canonical piped one-liner — still names the researcher, and passes the guard. Verified with scripts/check-style.sh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): fix the #458 red bats + 2 Bugbot bugs (newgidmap cap, write-failure) Root cause of the "540 ok but exit 1" bats red: the probe.bats uidmap tests set PATH="$bin" in the test body to hide system helpers, which also hides `rm` — so bats-core 1.10+ can't run its own per-test cleanup ("rm: command not found") and fails the whole run even though every test passes. Scope the hermetic PATH to a subshell so it can't leak into bats' machinery. (Why develop was green + this was so hard to see: these tests are new in slice 2, and the symptom is a clean pass list with a non-zero exit.) Two real Bugbot findings in the slice's own code: - _idmap_helper_ok checked cap_setuid for BOTH helpers; newgidmap carries cap_setgid (Arch filecaps) -> false-rejected. Map name->cap; fix the test mock that masked it + add a wrong-cap regression test. - _provision_subid_ranges printed success/returned 0 even when the usermod/tee write failed (callers run it with set -e off) -> installer proceeds with no range. Guard every write; warn + return 1 on failure. + a test. Verified: probe.bats + setup-linux.bats EXIT 0 (0 not-ok, 0 rm-not-found) in a faithful ubuntu 24.04 + bats 1.10 + non-root container. Rebased onto develop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): 2 more Bugbot findings on #458 (apt hang, false zero-root message) - _install_uidmap_pkg ran a bare `sudo apt-get install -y uidmap` under the spinner — no needrestart/DEBIAN_FRONTEND env, no DPkg::Lock::Timeout, no apt_wait_for_lock — so a headless Tier-1 install can hang on Ubuntu needrestart or an apt-daily lock (#210 class). Reuse the repo's hardened PM_INSTALL (populate via setup_pm, which Tier 1 skips) + apt_wait_for_lock. - install_rootless_docker always printed "no administrator rights were used", even after _ensure_subid_ranges performed an announced sudo touch on the root/sudo_nopw path. The gate now sets TB_ROOTLESS_ADMIN_TOUCH and the summary is honest on both the zero-root and one-admin-touch paths. Tests: hardened-install assertion (NEEDRESTART_MODE + DPkg::Lock::Timeout) + a success-message honesty test. Verified EXIT 0 (0 not-ok, 0 rm-errors) in the faithful ubuntu 24.04 + bats 1.10 + non-root container. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): refresh the package index in _install_uidmap_pkg (Bugbot #458) Completing the prior apt-hardening: _install_uidmap_pkg populated PM_INSTALL and waited for the dpkg lock but never ran PM_UPDATE. On the Tier-1 path this is the first package op, so an empty/stale index can't locate uidmap/shadow and the install hard-stops. Run $PM_UPDATE (best-effort) first, matching the repo's other install paths (setup-linux.sh:335/543). Test asserts the index refresh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(installer): silence the PS 5.1 progress throttle in install-k8s.ps1 downloads (#468 follow-up) (#471) Same class as the bootstrap fix in #469: PS 5.1's progress overlay throttles Invoke-WebRequest 10-50x and reads like a hang. One function-local $ProgressPreference in Invoke-WithRetry covers every fetch scriptblock it drives (dynamic scoping) - winget msixbundle, Docker Desktop fallback, kubectl, k3d, helm, GPU plugin yaml, and the version resolvers. Honest-progress expectation lines (sizes measured today via HEAD): Docker Desktop ~600 MB, winget ~200 MB, kubectl ~60 MB, k3d ~25 MB, helm ~20 MB - all cold-path only, silent on warm re-runs. Pester: 205 passed / 0 failed locally. PSSA: 0 errors. manifest.sha256 regenerated. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: shujaat hasan <shujaat@tracebloc.io> * fix(installer): trust the corporate CA in the Docker daemon for k3d runtime image pulls (#474) (#475) * fix(#474): detect host Docker daemon x509 at cluster-create + document daemon CA trust k3d pulls its own runtime images (rancher/k3s, k3d-tools, k3d-proxy) with the HOST Docker daemon, which doesn't use the in-node CA trust from #424. On a TLS-inspecting network that pull can x509-fail during 'k3d cluster create', before any node boots — so the post-create diagnosis never classifies it. - bash: _host_ca_create_hint() detects x509 in the k3d create output and prints a platform-aware remedy (Linux system trust store vs Docker Desktop VM); wired into _create_new_cluster's failure path. - PS: Write-HostCaCreateHint() mirrors it (Windows Trusted Root store), wired before the generic create failure. - docs/INSTALL.md: document trusting the CA in the daemon itself (Linux / Docker Desktop). - check-drift.sh: enforce both installers keep the host-CA hint (parity). - Tests: bats (Linux/macOS branches + silent-on-no-x509) + Pester + drift. Closes#474 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#474): distro-aware Linux CA remedy + actionable Docker Desktop for Linux (Bugbot) - Linux native-Docker remedy now covers both Debian/Ubuntu (update-ca-certificates) and RHEL/Fedora (update-ca-trust), not just the Debian path — the installer supports RHEL hosts where the Debian commands fail. - Docker Desktop for Linux now has an actionable step (trust in the system store, restart Docker Desktop) instead of a dangling reference to a step only printed on the macOS branch. - docs/INSTALL.md updated to match. bats Linux test asserts both distro paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#474): cover Colima runtime in the macOS host-CA remedy (Bugbot) Headless macOS installs use Colima (_install_docker_colima), a Lima VM that does not read the macOS keychain — so the 'trust it in the keychain + restart Docker Desktop' remedy was wrong for those hosts. The macOS branch now also gives the Colima path (add the CA inside the VM via 'colima ssh', then 'colima restart'). docs/INSTALL.md + macOS bats test updated to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(#474): isolate drift negative fixtures so a missing new token can't mask them (Bugbot) Adding _host_ca_create_hint / Write-HostCaCreateHint as required _drift_ca_trust tokens meant the older negative fixtures (missing registry-config, comment-only registry-config) could pass just because the new token was also absent — so the comment-strip case no longer uniquely proved comment-stripping still works. Each negative fixture now carries ALL other required tokens and omits/comments only the one under test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#474): surface host-CA hint on the PS create-timeout path too (Bugbot parity) The PowerShell create-timeout branch exited via Err without calling Write-HostCaCreateHint (and deleted the k3d logs first), so a TLS-inspected host pull that logs x509 then hangs to the deadline gave Windows operators a raw timeout with no certlm.msc CA guidance — while bash runs _host_ca_create_hint on its timeout fall-through. Capture the full create output before deleting the logs and call the hint before the timeout Err. Adds a parity regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#474): use a herestring in _host_ca_create_hint to survive pipefail (Asad) printf '%s' "$out" | grep -qiE ... could swallow the hint under set -o pipefail: grep -q closes the pipe on first match, so for output past the ~64KB pipe buffer (reachable on the timeout path, which passes the full logs) printf takes SIGPIPE, the pipeline exits non-zero, and `|| return 0` bails even though x509 matched. Feed grep via a herestring (no pipe, no SIGPIPE). Adds a >64KB-under-pipefail regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(install): Tier-1 k3d-on-rootless + cgroup delegation + autostart (#1221) (#478) * feat(install): Tier-1 k3d-on-rootless + cgroup delegation + autostart (#1221) Slice #1221 (RFC 0001 / #1177): make a rootless Tier-1 cluster actually usable, all behind the opt-in TB_TIER1_ROOTLESS flag (default off until the spike's §5 host validation). With the flag unset every path below is a no-op and current behavior is byte-for-byte unchanged. - Shared _rootless_active predicate (common.sh) so cluster.sh + setup-linux.sh can't drift on the flag pair. - create_cluster targets the rootless socket (DOCKER_HOST); ensure_cluster_ autostart gets a user-scope branch (systemctl --user enable + loginctl enable-linger, never `sudo systemctl enable docker`), and promises reboot-survival only when BOTH succeed (honesty rule, #375/#458). - cgroup v2 controller delegation drop-in (Delegate=cpu cpuset io memory pids): privileged write + daemon-reload on root/sudo, or hand off to prepare-host with the exact path+content when unprivileged. run_prepare_host writes it too (system-wide -> covers the researcher). - Carry-ins from #452/#458: scope-aware _configure_docker_proxy (user scope, no sudo) so a proxy-only host's rootless daemon can pull rancher/k3s; _set_tools_target installs user-space on rootless Tier 1 (no sudo-mv crash on a true no-sudo host); persist DOCKER_HOST to the shell rc for new terminals. 14 new bats tests incl. flag-off regressions; shellcheck --severity=error clean; manifest.sha256 regenerated (R8). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): address Bugbot on #478 — rootless autostart seed + admin-touch msg - ensure_cluster_autostart: don't seed TB_DOCKER_AUTOSTART from the SYSTEM docker.service is-enabled check on the rootless path. The cluster runs on the per-user rootless socket, so a system unit that happens to be enabled would seed a false reboot promise the rootless branch then can't honestly retract. On rootless the user-scope enable+linger are now the sole authority (Bugbot medium). - install_rootless_docker: the TB_ROOTLESS_ADMIN_TOUCH success line no longer hardcodes "subuid/subgid range" — _ensure_cgroup_delegation can set that flag too, so it now names "host prerequisites (subuid/subgid range and/or cgroup delegation)" (Bugbot low). - Test: rootless + system docker.service enabled + user-enable fails => flag stays 0 (pins the seed-guard). manifest regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): address Asad + Saqlain review on #478 - _persist_docker_host: key idempotency off our own marker, not a bare 'DOCKER_HOST=' probe. The old probe also matched a user's own DOCKER_HOST (remote/TCP), so we silently skipped persisting the rootless socket and new shells kept hitting the wrong daemon. Now: our own line -> idempotent; a foreign DOCKER_HOST -> left untouched + a warn to repoint it (Asad #2 + Bugbot #478, Medium). - ensure_cluster_autostart: reset TB_DOCKER_AUTOSTART=0 in the rootless else-branch (defensive; the is-enabled seed is already guarded off the rootless path) so the honesty guarantee is local to the branch (Asad #1). - install_rootless_docker: success line now reads "one or more one-time admin steps" so it doesn't undercount when both the subuid and cgroup touches happen (Saqlain #1). - Test: foreign DOCKER_HOST -> warns, no clobber, no double-write. manifest regen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(installer): failures surface the real error + log path + support-bundle hint (#423) (#476) * fix(#423): failures surface the real error + log path + support-bundle hint Fatal errors printed a generic red line while the actionable detail (k3d/helm stderr) went only to the transcript, and the log path itself was never shown on screen. Now: - Err gains an optional $Detail param; Get-ErrDetailLines (pure, unit-tested) renders the last ~5 non-empty output lines + the log path + a '-Diagnose' next-step hint, appended to EVERY fatal error. - Cluster-create failure passes k3d's stdout/stderr so the real reason (image pull / proxy / port / WSL) shows on screen — the motivating case. - Helm repo-add / reconcile / install failures pass helm's output via $Detail instead of embedding it (no more duplicated log-path text). - Install log path is announced up front in the banner (was log-only before). Closes#423 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#423): force array enumeration of Err detail lines (Bugbot, defensive) Bugbot flagged that a single-line Get-ErrDetailLines return (no detail + no LOG_FILE, e.g. a Confirm-Config failure before Start-InstallLog) unwraps to a scalar string. The foreach statement already iterates a scalar once (verified: it prints the whole line, not per-character), so the reported char-splitting does not reproduce -- but wrap the enumeration in @(...) to make that unambiguous and future-proof. Adds a regression test asserting the single-line case stays one intact line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#423): strip PS 5.1 ErrorRecord chrome from the failure excerpt (Bugbot) helm failures arrive as `native 2>&1 | Out-String`; on Windows PowerShell 5.1 that wraps stderr in ErrorRecord chrome (the `At <file>:<n> char:<n>` position line plus the `+ ...` / `+ CategoryInfo` / `+ FullyQualifiedErrorId` block). Get-ErrDetailLines kept only the last 5 non-empty lines, so the excerpt was all chrome and the real `Error:` line dropped out -- a regression from the previous full-message dump. Filter those chrome lines before taking the window so the actual error survives. Adds a regression test simulating the 5.1 rendering. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#423): drop duplicate inline log-path hints (Bugbot) Err now always prints the log path via Get-ErrDetailLines, so the k3d spawn-failure and create-timeout paths that still Hint "Full log:" right before Err printed it twice. Remove those inline hints; Err is the single source. Adds a guard test asserting no inline 'Full log:' hints remain in the installer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#423): put stderr last in the create-failure Err detail (Asad) Get-ErrDetailLines keeps the LAST 5 non-empty lines, so with detail ordered stderr-then-stdout any k3d stdout tail could crowd the real stderr reason (FATA/x509/port) out of the excerpt. Order it stdout-then-stderr so the stderr tail survives the window; also matches the Write-HostCaCreateHint order just above. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Syed Is Saqlain <saqlain.syed007@gmail.com> Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Arturo Peroni <arturo@tracebloc.io> Co-authored-by: shujaat_tracebloc <153823837+shujaatTracebloc@users.noreply.github.com> Co-authored-by: shujaat hasan <shujaat@tracebloc.io> Co-authored-by: tracebloc-release-train[bot] <309815517+tracebloc-release-train[bot]@users.noreply.github.com>
8 tasks
LukasWodka added a commit
that referenced
this pull request
Aug 15, 2026
…pying it (#723) * release-train: staging -> main (#495) * chore(chart): close values-schema gaps + drop dead override/code/docs (#963) (#457) * chore(chart): close values-schema gaps + drop dead override/code/docs (#963) Contract fixes for the client Helm chart (re-verified against develop at chart v1.9.6; the #963 audit was taken at v1.8.4): - values.schema.json: add the six live-but-unvalidated keys so bad values fail `helm lint` instead of silently passing — egressReachabilityCheck.enabled, ingestionAuthz.{allowed,serviceAccountName}, networkPolicy.training.enforcementProbeTimeoutSeconds, podTokenSigningSecret, podTokenTtlSeconds. Types/defaults/constraints taken from values.yaml and the templates that consume them. helm lint passes. - ingestor subchart: remove the dead `image.repository` key — no template ever rendered it (jobs-manager spawns from the parent chart's images.ingestor.repository). Kept image.digest (live). README's air-gapped override rows now point at the authoritative parent-chart path. - README: drop the hardcoded chart version (said v1.3.5 while Chart.yaml is 1.9.6) and point to Chart.yaml / the releases page, so it can't drift again. - Delete the unwired check_docker_arch_mac function + its bats test (no call sites) and the orphaned docs/eks.md (referenced nowhere). Part of tracebloc/backend#963. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(installer): regenerate manifest after common.sh trim + develop merge The #963 chart-contract cleanup dropped 48 dead lines from scripts/lib/common.sh, changing its sha; the installer manifest wasn't regenerated, so the Static analysis gate (gen-manifest.sh --check) failed. Merging develop also refreshed preflight.sh/install-k8s.ps1 hashes. Regenerate scripts/manifest.sha256 to match the working tree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(release): package charts as the tag version + pre-releases skip gh-pages (#467) * fix(release): package charts AS the tag version + pre-releases skip gh-pages Incident 2026-07-29: the v1.9.7-rc.1 pre-release packaged the client chart from Chart.yaml's plain 1.9.7 and pushed it into the public helm index as a STABLE version -- customers running helm upgrade would have received staging content (removed from the index by hand, tgz deleted). Two layers now prevent it: (1) helm package --version/--app-version from the release tag, so rc charts carry the -rc.N suffix helm's pre-release rules key on; (2) pre-releases never run the gh-pages index steps at all -- FR consumes the release assets (stamped installer / chart tgz), the index is a customer surface reserved for finals. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gh-pages gates key on verify's tag-derived prerelease, not the frozen event (Bugbot) github.event.release.prerelease is an event-time snapshot: after verify demotes a mis-marked release, it still reads false, so the demoted rc would have entered the public index anyway. verify now outputs effective prerelease-ness derived from the tag shape (the same strict rule the demotion uses) and all three gh-pages steps gate on that output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: post-publish index-invariant job (manual leak catch -> CI) After every release run: the public index must contain only stable-shaped versions, and a prerelease run must not have indexed its own version. Fails loudly; would have caught the 1.9.7 leak within a minute of it happening instead of during manual FR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(install): rootless Docker core + Tier-1 routing (opt-in) (#1219) (#452) * feat(install): rootless Docker core + Tier-1 routing (opt-in) Add install_rootless_docker() and a Tier-1 early-branch in install_linux so a modern-kernel host with no runtime and no root can install entirely in user space (RFC 0001 Tier 1 — the RFC's primary path). Gated behind opt-in TB_TIER1_ROOTLESS=1; with the flag unset a Tier-1 host falls through to the legacy privileged flow unchanged (validated default). - install_rootless_docker: uidmap-helper precondition (defers to prepare-host #1178 when absent — never self-sudo), no-sudo install via dockerd-rootless-setuptool.sh or get.docker.com/rootless, user-scoped systemctl --user + loginctl enable-linger, DOCKER_HOST export with XDG_RUNTIME_DIR fallback, single docker-info verify (no retry loop). - Tier-1 branch mirrors the Tier-0 early-return. Tools still install via sudo here (_set_tools_target keys no-sudo off Tier 0 only) — tightening that for rootless Tier 1 is deferred to slice 3 (#1221). - 6 bats cases; scripts/manifest.sha256 regenerated (R8). Closestracebloc/backend#1219 Part of tracebloc/backend#1177 · Epic tracebloc/backend#1168 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address Bugbot review on rootless Docker core (#452) - Prepend ~/bin to PATH after the rootless install so this run's docker info verify + later k3d/docker calls resolve the CLI the get.docker.com/rootless fallback installs there (High). - Bound the rootless `docker info` verify with a new shared _bounded helper (timeout/gtimeout, mirrors probe.sh) so a wedged user daemon can't hang a headless install (Medium). - Guard the user-systemd bring-up under set -e: `systemctl --user … || true` (the bounded verify is the real gate) and `loginctl enable-linger … || warn` (optional; fails on polkit-locked hosts even when the daemon is up) (Medium). Adds 2 bats cases (~/bin on PATH; systemd/linger failure falls through to the verify). Manifest regenerated (R8). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: correct the uidmap remedy message (Bugbot #452) The missing-uidmap error claimed prepare-host would install the uidmap helpers, but run_prepare_host only sets up privileged Docker + the docker group — it never installs uidmap. Point at the two honest remedies instead: install the `uidmap` package directly (rootless then works), or run prepare-host to set up Docker so the researcher installs at Tier 0 (no rootless needed). #1220 folds this into the shared subuid/subgid gate and teaches prepare-host to install uidmap for real. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(install): TODO(#1221) — rootless daemon needs user-scoped proxy config Bugbot on #452 flagged that install_rootless_docker never configures a corporate proxy for the user-scoped dockerd (the #244 _configure_docker_proxy is sudo/system-scoped and the Tier-1 early-return never reaches it), so k3d pulls of rancher/k3s time out on proxy-only hosts. Deferred to #1221 (the k3d-on-rootless-socket slice that owns the pulls); leaving a tracked TODO so the follow-up adds the user-scoped drop-in. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address Asad review nits on #452 — chmod no-op + misleading comment - Drop the chmod +x on the rootless installer script: it runs via `sh "$rootless_script"`, which ignores the exec bit. - Reword the Tier-1 _install_userspace_tools comment: tools still sudo-install on Tier 1 (only _persist_tools_on_path is no-sudo); the comment previously implied otherwise. The underlying _set_tools_target sudo-crash on no-sudo hosts and the post-install DOCKER_HOST shell persistence are tracked to #1221. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: resolve user via id -un in install_rootless_docker (Saqlain review, #452) $USER can be empty in headless / su / cron contexts (a Tier-1 target), which would break `loginctl enable-linger` and the success line. Resolve the user once via `id -un` (fallback $USER) and use it for the linger call, its hint, and the success message. Matches the id-based robustness DOCKER_HOST already uses. Happy-path bats now mocks `id -un` cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: add code-quality caller workflow (advisory) (#463) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * Add pre-commit hooks (Layer 0, lint-only) (#465) * Add pre-commit config (Layer 0, lint-only) Lint-only on purpose: scripts/manifest.sha256 must keep matching the bytes under scripts/, so no hook may rewrite files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Document pre-commit setup in README Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * docs: add Bugbot resolve-and-reply team norm to .cursor/BUGBOT.md (#464) Part of tracebloc/backend#1308 Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * ci: cover scripts/resolve-ingestor-digest.sh in CI shellcheck (#466) * ci: lint scripts/resolve-ingestor-digest.sh in CI shellcheck (was never linted) Both CI shellcheck invocations enumerate files explicitly and both omitted this script. Verified clean against shellcheck --severity=error --shell=bash 0.11.0 before adding. The pre-commit hook from #465 already covers it locally; this closes the same gap on the CI side. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci: lint scripts/resolve-ingestor-digest.sh in CI shellcheck (was never linted) Both CI shellcheck invocations enumerate files explicitly and both omitted this script. Verified clean against shellcheck --severity=error --shell=bash 0.11.0 before adding. The pre-commit hook from #465 already covers it locally; this closes the same gap on the CI side. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): honest cosign bootstrap download + translated DISM failures (#468) (#469) The v1.9.7-rc.1 FR killed a healthy install: PS 5.1's progress overlay throttled the 17 MB pinned-cosign fetch to ~4.5 min of dead silence and the window read as frozen. - silence the PS 5.1 progress overlay in Get-WithRetry/Get-Optional (function-local, auto-reverts) - the classic 10-50x IWR speedup - run the cosign fetch in a background job with a dim liveness tick (Wait-JobWithTicks / Get-OptionalWithTicks; cwd pinned per #409, TLS 1.2 re-applied in the fresh process), expectation lines before, elapsed + checksum-verified confirmation after - ASCII-only string literals in both installers: the release asset is served without a charset so PS 5.1's irm decodes UTF-8 source as Latin-1 before iex, and BOM-less -File reads are ANSI - literal em-dashes/ellipses reached customers as mojibake. Locked in by a tokenizer-based Pester test (which also caught the -Help here-string). - Enable-OneVirtFeature: translate DISM's raw COMException (feature package absent on Server SKUs vs enable failure) and stop demanding a reboot for a feature that never enabled (old code sent Server users into a reboot->re-run->same-error loop) Pester: 212 passed / 0 failed locally (pwsh 7.5, Pester 5.7.1). PSScriptAnalyzer: 0 errors. manifest.sha256 regenerated. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * fix(installer): trust the corporate MITM CA in the k3d nodes (#424) (#453) * fix(installer): trust the corporate MITM CA in the k3d nodes (#424) Proxy REACHABILITY reaches the nodes, but on a TLS-inspecting (break-and- inspect) network the nodes still don't TRUST the corporate CA, so every in-node containerd pull (rancher/k3s, ghcr.io, tracebloc images) fails x509 — then masked (helm runs without --wait) into a root-cause-free "an image couldn't be pulled." Enterprise/hospital archetype, all three OSes. - Inject the CA at create time: when TRACEBLOC_CA_BUNDLE (or CURL_CA_BUNDLE) is set, mount the bundle into every k3d node and write a registries.yaml pointing containerd at it per-registry (docker.io, registry-1.docker.io, ghcr.io), via the same --config/create path that already carries proxy env. Parity across scripts/lib/cluster.sh (Linux/macOS) and install-k8s.ps1 (Windows). A CA var set but unreadable fails loudly instead of silently skipping. - Name the env var where the user hits the wall: the TLS-interception preflight hint (both OSes), docs/INSTALL.md, and the PS -Help env-var list. - CA-aware diagnosis: detect x509 / "certificate signed by unknown authority" pull events and report a dedicated image_pull_ca state — "the cluster does not trust your network's TLS-inspection CA" + the exact remedy — instead of the generic pull error. Mirrored in summary.sh and Print-Summary. - New check-drift.sh parity check (_drift_ca_trust) so neither installer can drop the CA wiring for the other's OS. Tests: +8 cluster.bats, +3 summary.bats, +2 check-drift.bats, +8 Pester. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): CA-trust hardening — no fail-open, bounded events, verify CA readable (Bugbot #424) Three Bugbot findings on #424: - _write_k3d_registries_config failed open: on mktemp failure it returned success with no path, so create still mounted the CA and logged "nodes trust it" but dropped --registry-config → containerd never got ca_file, x509 pulls still fail while the operator thinks it's fixed. Now returns non-zero; the caller hard-errors (CA was supplied, so we refuse to proceed without wiring it in). - PS Get-NotReadyState `kubectl get events` had no --request-timeout (the bash path does) — on a wedged/proxy-misrouted API, classification could hang. Added --request-timeout=5s to match _diagnose_not_ready. - PS Resolve-CaBundle only checked existence (Test-Path), not readability, so an unreadable CA passed on Windows but bash (-r) hard-fails. Added an OpenRead probe so both fail the same way, up front. Tests: cluster.bats +mktemp-failure + unwritable-registries-hard-error; install-k8s.Tests.ps1 +unreadable-CA (Unix) + events --request-timeout assertion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): errexit-safe CA-resolve capture + drift check ignores comments (Bugbot #424 r2) Two round-2 Bugbot findings: - Under `set -euo pipefail`, `ca_bundle="$(_resolve_ca_bundle)"; ca_rc=$?` exited on the rc-2 (unreadable/missing CA) BEFORE ca_rc/error ran — operators got a bare exit instead of the "can't be read" guidance. Capture with `|| ca_rc=$?` so errexit doesn't fire and the guidance prints. - _drift_ca_trust whole-file grep matched tokens in comments (e.g. --registry-config appears in a comment above the real line), so deleting the functional wiring could still pass. Strip comment lines first (matches the execute-gate / preflight-host checks), no grep -q under pipefail. Tests: cluster.bats +errexit-safe-capture; check-drift.bats +comment-only-token drift. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): TLS-preflight hint names the right var per layer/OS (Bugbot #424 r3) The hint claimed TRACEBLOC_CA_BUNDLE makes "the host AND the k3d nodes" trust the CA, but the host connectivity checks use curl_secure / Invoke-WebRequest, which read CURL_CA_BUNDLE / the system trust store — not TRACEBLOC_CA_BUNDLE (that var only reaches the nodes via _resolve_ca_bundle). Following the hint literally left host preflight TLS failures unchanged. Corrected, no behaviour change: - bash: CURL_CA_BUNDLE fixes these host checks AND the nodes; TRACEBLOC_CA_BUNDLE is nodes-only; or add the CA to the system trust store. - Windows: import the CA into the cert store for the host checks (Invoke-WebRequest uses the store, not an env var); TRACEBLOC_CA_BUNDLE/CURL_CA_BUNDLE cover the nodes. (Reworded to avoid a bare lowercase `curl` that the curl_secure style guard flags.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): apply CA on cluster REUSE path — warn + recreate guidance (Bugbot #424 r4) The image_pull_ca remedy said "set the CA and re-run", but CA trust is baked in only at fresh create; a re-run reuses the existing cluster and never mounts the CA or passes --registry-config, so the x509 pulls persisted. Mirror the existing proxy handling (baked-at-create → warn on reuse): - bash _check_existing_cluster_ca (called from _handle_existing_cluster): warns when a CA bundle is set but the reused server container lacks the CA mount. - ps1 New-K3dCluster reuse block: same check via docker inspect mounts. - both image_pull_ca remedies now say to `k3d cluster delete <name>` first, then re-run with the CA (CA, like proxy, can't be added to a running cluster). Tests: cluster.bats +3 (no-CA no-op / CA-but-missing-mount warns / mount-present silent). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): add auth.docker.io to the CA registries config (Bugbot #424 r5) The registries.yaml ca_file entries covered docker.io / registry-1.docker.io / ghcr.io, but Docker Hub pulls also TLS-handshake with auth.docker.io for bearer tokens — so on a break-and-inspect network containerd still rejected the intercepted cert there even with the CA mounted. #416 already probes auth.docker.io at preflight; the CA registries list now matches. Added to TB_CA_REGISTRIES and $TbCaRegistries; registries.yaml test counts 3 -> 4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#424): _resolve_ca_bundle rejects a directory, not just unreadable paths A directory of PEMs is readable (-r) but would bind-mount over the single node ca_file path and containerd can't read it — the silent 'looks applied but still x509' case. Require a regular file (-f), mirroring the PS Resolve-CaBundle -PathType Leaf check. Adds a directory-reject bats case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#424): exact whole-line match for CA mount detection (Bugbot) _check_existing_cluster_ca used a substring test on docker mount destinations, so a longer path embedding /etc/ssl/certs/tracebloc-mitm-ca.crt (e.g. …crt.bak) would be treated as the CA mount and skip the recreate warning while containerd still x509-fails. Switch to grep -qxF (exact whole-line), matching the PS anchored regex. Adds a substring-embed test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#424): scope x509 classification to the pull-failure event (Asad) _diagnose_not_ready / Get-NotReadyState flagged image_pull_ca on ANY x509 event in the namespace, so a stale/unrelated x509 event (e.g. a FailedMount) could misdirect the user into a needless delete+recreate. Filter events to the image-pull failure lines (failed to pull / ErrImagePull) before testing x509, in both bash and PS. Adds an unrelated-x509 test to each side. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(chart): perIngestionTables — RFC-0003 D16 enablement knob (backend#1205) (#472) * feat(chart): perIngestionTables — the RFC-0003 D16 enablement knob (backend#1204/#1205) values.perIngestionTables (default false, schema-typed) renders PER_INGESTION_TABLES=1 onto the jobs-manager, which forwards it into every ingestion Job it spawns (client-runtime companion PR). Flip per environment, dev first, only once that environment's backend + engine images + jobs-manager carry the merged D-series. Default installs render byte-identically (conditional block; unit tests pin both sides). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(values): own banner for perIngestionTables — it is not part of the authz section (review) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * chore: clear house-rules findings (#470) Fix every finding the shared org checker (tracebloc/.github scripts/house-rules.sh) reports at develop HEAD: missing curl timeouts/TLS floors, plus (cli) a missing pipefail. Waivers only where the finding is a documented false positive. Part of tracebloc/backend#1303. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * feat(install): Tier-1 subuid/subgid gate + prepare-host remediation (#1220) (#458) * feat(install): Tier-1 subuid/subgid gate + prepare-host remediation RFC 0001 #1220. Detect the one privileged residue a modern rootless host may still need — a subordinate UID/GID range + the setuid uidmap helpers — and either proceed (present), hand off to prepare-host (unprivileged), or perform one announced touch (sudo available). Never blanket sudo, never an opaque mid-install crash inside dockerd-rootless-setuptool.sh. - probe.sh: _probe_subid_ranges (PROBE_SUBID) + _probe_uidmap_helpers (PROBE_UIDMAP), set in run_host_probes (Linux only), plus audit rows on the Tier-1 path. - common.sh: shared pure parsers _subid_has_entry + _next_subid_start, used by both the probe and the remediation (no duplication). - setup-linux.sh: _ensure_subid_ranges gate (present / hand-off / one announced sudo touch) called before install_rootless_docker; _provision_subid_ranges (idempotent, non-overlapping block, usermod --add-subuids with file-append fallback, uidmap install) shared by the installer and run_prepare_host. Folds in slice-1's minimal uidmap check. - Tests: probe.bats + setup-linux.bats. Manifest regenerated (R8). Closestracebloc/backend#1220 Part of tracebloc/backend#1177 · Epic tracebloc/backend#1168 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): address #458 review — subid gate/probe/hand-off robustness Bugbot + @saadqbal + a self code-review on client#458, all in slice-2's code: - id -un everywhere (gate, _provision default, probe): $USER diverges from the rootless daemon's user under su/cron, which wedged detection/provisioning (#1). - Re-verify the uidmap helpers are usable (present AND setuid|cap_setuid) after install, and return non-zero + warn (NOT error/exit) so run_prepare_host stays best-effort while the installer sudo-path hard-fails via `|| error` (#2 + self-review). - _idmap_helper_ok (common.sh): accept the setuid bit OR a cap_setuid filecap, so Arch's `shadow`/pacman path isn't false-rejected (#3). - Hand-off + run_prepare_host fallback compute a non-overlapping start via _next_subid_start (honoring TB_SUBUID_FILE/TB_SUBGID_FILE), not hardcoded 100000 (#4 + self-review path-override). - Hand-off command names the researcher (TB_PREPARE_USER=) — bare prepare-host provisions nothing, so it would have looped back to the same hand-off (#5). - Capture `usermod --help` before grepping — pipefail-safe (#6). bats: id -un mocks, filecaps accept/reject, gate hand-off (names user + computed start), _provision re-verify best-effort, run_prepare_host best-effort. R8 regen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(install): stub the subid gate in the Tier-1 rootless routing test install_linux's Tier-1 branch now calls _ensure_subid_ranges (slice 2) before install_rootless_docker; the routing test left it un-stubbed, so the real gate hit the no-sudo hand-off and error()'d → install_linux returned non-zero. Stub _ensure_subid_ranges (its own behavior is covered by the dedicated gate tests) and assert it runs before daemon setup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(install): scope set -o pipefail to a subshell (bats harness footgun) Setting `set -o pipefail` in the @test body can leak into bats' own post-test pipelines and fail the whole run with exit 1 even when every test reports ok (no 'not ok'). Confine it to a subshell around the call so the pipefail-safety assertion still holds without touching the harness. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): style guard — no bare curl in the prepare-host hint The hand-off piped 'curl … | TB_PREPARE_USER=… bash', which breaks check-style.sh's exemption for the canonical 'curl … | bash' one-liner (the env var sits between the pipe and bash). Split into an 'export TB_PREPARE_USER=…' line + the canonical piped one-liner — still names the researcher, and passes the guard. Verified with scripts/check-style.sh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): fix the #458 red bats + 2 Bugbot bugs (newgidmap cap, write-failure) Root cause of the "540 ok but exit 1" bats red: the probe.bats uidmap tests set PATH="$bin" in the test body to hide system helpers, which also hides `rm` — so bats-core 1.10+ can't run its own per-test cleanup ("rm: command not found") and fails the whole run even though every test passes. Scope the hermetic PATH to a subshell so it can't leak into bats' machinery. (Why develop was green + this was so hard to see: these tests are new in slice 2, and the symptom is a clean pass list with a non-zero exit.) Two real Bugbot findings in the slice's own code: - _idmap_helper_ok checked cap_setuid for BOTH helpers; newgidmap carries cap_setgid (Arch filecaps) -> false-rejected. Map name->cap; fix the test mock that masked it + add a wrong-cap regression test. - _provision_subid_ranges printed success/returned 0 even when the usermod/tee write failed (callers run it with set -e off) -> installer proceeds with no range. Guard every write; warn + return 1 on failure. + a test. Verified: probe.bats + setup-linux.bats EXIT 0 (0 not-ok, 0 rm-not-found) in a faithful ubuntu 24.04 + bats 1.10 + non-root container. Rebased onto develop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): 2 more Bugbot findings on #458 (apt hang, false zero-root message) - _install_uidmap_pkg ran a bare `sudo apt-get install -y uidmap` under the spinner — no needrestart/DEBIAN_FRONTEND env, no DPkg::Lock::Timeout, no apt_wait_for_lock — so a headless Tier-1 install can hang on Ubuntu needrestart or an apt-daily lock (#210 class). Reuse the repo's hardened PM_INSTALL (populate via setup_pm, which Tier 1 skips) + apt_wait_for_lock. - install_rootless_docker always printed "no administrator rights were used", even after _ensure_subid_ranges performed an announced sudo touch on the root/sudo_nopw path. The gate now sets TB_ROOTLESS_ADMIN_TOUCH and the summary is honest on both the zero-root and one-admin-touch paths. Tests: hardened-install assertion (NEEDRESTART_MODE + DPkg::Lock::Timeout) + a success-message honesty test. Verified EXIT 0 (0 not-ok, 0 rm-errors) in the faithful ubuntu 24.04 + bats 1.10 + non-root container. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): refresh the package index in _install_uidmap_pkg (Bugbot #458) Completing the prior apt-hardening: _install_uidmap_pkg populated PM_INSTALL and waited for the dpkg lock but never ran PM_UPDATE. On the Tier-1 path this is the first package op, so an empty/stale index can't locate uidmap/shadow and the install hard-stops. Run $PM_UPDATE (best-effort) first, matching the repo's other install paths (setup-linux.sh:335/543). Test asserts the index refresh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(installer): silence the PS 5.1 progress throttle in install-k8s.ps1 downloads (#468 follow-up) (#471) Same class as the bootstrap fix in #469: PS 5.1's progress overlay throttles Invoke-WebRequest 10-50x and reads like a hang. One function-local $ProgressPreference in Invoke-WithRetry covers every fetch scriptblock it drives (dynamic scoping) - winget msixbundle, Docker Desktop fallback, kubectl, k3d, helm, GPU plugin yaml, and the version resolvers. Honest-progress expectation lines (sizes measured today via HEAD): Docker Desktop ~600 MB, winget ~200 MB, kubectl ~60 MB, k3d ~25 MB, helm ~20 MB - all cold-path only, silent on warm re-runs. Pester: 205 passed / 0 failed locally. PSSA: 0 errors. manifest.sha256 regenerated. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: shujaat hasan <shujaat@tracebloc.io> * fix(installer): trust the corporate CA in the Docker daemon for k3d runtime image pulls (#474) (#475) * fix(#474): detect host Docker daemon x509 at cluster-create + document daemon CA trust k3d pulls its own runtime images (rancher/k3s, k3d-tools, k3d-proxy) with the HOST Docker daemon, which doesn't use the in-node CA trust from #424. On a TLS-inspecting network that pull can x509-fail during 'k3d cluster create', before any node boots — so the post-create diagnosis never classifies it. - bash: _host_ca_create_hint() detects x509 in the k3d create output and prints a platform-aware remedy (Linux system trust store vs Docker Desktop VM); wired into _create_new_cluster's failure path. - PS: Write-HostCaCreateHint() mirrors it (Windows Trusted Root store), wired before the generic create failure. - docs/INSTALL.md: document trusting the CA in the daemon itself (Linux / Docker Desktop). - check-drift.sh: enforce both installers keep the host-CA hint (parity). - Tests: bats (Linux/macOS branches + silent-on-no-x509) + Pester + drift. Closes#474 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#474): distro-aware Linux CA remedy + actionable Docker Desktop for Linux (Bugbot) - Linux native-Docker remedy now covers both Debian/Ubuntu (update-ca-certificates) and RHEL/Fedora (update-ca-trust), not just the Debian path — the installer supports RHEL hosts where the Debian commands fail. - Docker Desktop for Linux now has an actionable step (trust in the system store, restart Docker Desktop) instead of a dangling reference to a step only printed on the macOS branch. - docs/INSTALL.md updated to match. bats Linux test asserts both distro paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#474): cover Colima runtime in the macOS host-CA remedy (Bugbot) Headless macOS installs use Colima (_install_docker_colima), a Lima VM that does not read the macOS keychain — so the 'trust it in the keychain + restart Docker Desktop' remedy was wrong for those hosts. The macOS branch now also gives the Colima path (add the CA inside the VM via 'colima ssh', then 'colima restart'). docs/INSTALL.md + macOS bats test updated to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(#474): isolate drift negative fixtures so a missing new token can't mask them (Bugbot) Adding _host_ca_create_hint / Write-HostCaCreateHint as required _drift_ca_trust tokens meant the older negative fixtures (missing registry-config, comment-only registry-config) could pass just because the new token was also absent — so the comment-strip case no longer uniquely proved comment-stripping still works. Each negative fixture now carries ALL other required tokens and omits/comments only the one under test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#474): surface host-CA hint on the PS create-timeout path too (Bugbot parity) The PowerShell create-timeout branch exited via Err without calling Write-HostCaCreateHint (and deleted the k3d logs first), so a TLS-inspected host pull that logs x509 then hangs to the deadline gave Windows operators a raw timeout with no certlm.msc CA guidance — while bash runs _host_ca_create_hint on its timeout fall-through. Capture the full create output before deleting the logs and call the hint before the timeout Err. Adds a parity regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#474): use a herestring in _host_ca_create_hint to survive pipefail (Asad) printf '%s' "$out" | grep -qiE ... could swallow the hint under set -o pipefail: grep -q closes the pipe on first match, so for output past the ~64KB pipe buffer (reachable on the timeout path, which passes the full logs) printf takes SIGPIPE, the pipeline exits non-zero, and `|| return 0` bails even though x509 matched. Feed grep via a herestring (no pipe, no SIGPIPE). Adds a >64KB-under-pipefail regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(install): Tier-1 k3d-on-rootless + cgroup delegation + autostart (#1221) (#478) * feat(install): Tier-1 k3d-on-rootless + cgroup delegation + autostart (#1221) Slice #1221 (RFC 0001 / #1177): make a rootless Tier-1 cluster actually usable, all behind the opt-in TB_TIER1_ROOTLESS flag (default off until the spike's §5 host validation). With the flag unset every path below is a no-op and current behavior is byte-for-byte unchanged. - Shared _rootless_active predicate (common.sh) so cluster.sh + setup-linux.sh can't drift on the flag pair. - create_cluster targets the rootless socket (DOCKER_HOST); ensure_cluster_ autostart gets a user-scope branch (systemctl --user enable + loginctl enable-linger, never `sudo systemctl enable docker`), and promises reboot-survival only when BOTH succeed (honesty rule, #375/#458). - cgroup v2 controller delegation drop-in (Delegate=cpu cpuset io memory pids): privileged write + daemon-reload on root/sudo, or hand off to prepare-host with the exact path+content when unprivileged. run_prepare_host writes it too (system-wide -> covers the researcher). - Carry-ins from #452/#458: scope-aware _configure_docker_proxy (user scope, no sudo) so a proxy-only host's rootless daemon can pull rancher/k3s; _set_tools_target installs user-space on rootless Tier 1 (no sudo-mv crash on a true no-sudo host); persist DOCKER_HOST to the shell rc for new terminals. 14 new bats tests incl. flag-off regressions; shellcheck --severity=error clean; manifest.sha256 regenerated (R8). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): address Bugbot on #478 — rootless autostart seed + admin-touch msg - ensure_cluster_autostart: don't seed TB_DOCKER_AUTOSTART from the SYSTEM docker.service is-enabled check on the rootless path. The cluster runs on the per-user rootless socket, so a system unit that happens to be enabled would seed a false reboot promise the rootless branch then can't honestly retract. On rootless the user-scope enable+linger are now the sole authority (Bugbot medium). - install_rootless_docker: the TB_ROOTLESS_ADMIN_TOUCH success line no longer hardcodes "subuid/subgid range" — _ensure_cgroup_delegation can set that flag too, so it now names "host prerequisites (subuid/subgid range and/or cgroup delegation)" (Bugbot low). - Test: rootless + system docker.service enabled + user-enable fails => flag stays 0 (pins the seed-guard). manifest regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(install): address Asad + Saqlain review on #478 - _persist_docker_host: key idempotency off our own marker, not a bare 'DOCKER_HOST=' probe. The old probe also matched a user's own DOCKER_HOST (remote/TCP), so we silently skipped persisting the rootless socket and new shells kept hitting the wrong daemon. Now: our own line -> idempotent; a foreign DOCKER_HOST -> left untouched + a warn to repoint it (Asad #2 + Bugbot #478, Medium). - ensure_cluster_autostart: reset TB_DOCKER_AUTOSTART=0 in the rootless else-branch (defensive; the is-enabled seed is already guarded off the rootless path) so the honesty guarantee is local to the branch (Asad #1). - install_rootless_docker: success line now reads "one or more one-time admin steps" so it doesn't undercount when both the subuid and cgroup touches happen (Saqlain #1). - Test: foreign DOCKER_HOST -> warns, no clobber, no double-write. manifest regen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(installer): failures surface the real error + log path + support-bundle hint (#423) (#476) * fix(#423): failures surface the real error + log path + support-bundle hint Fatal errors printed a generic red line while the actionable detail (k3d/helm stderr) went only to the transcript, and the log path itself was never shown on screen. Now: - Err gains an optional $Detail param; Get-ErrDetailLines (pure, unit-tested) renders the last ~5 non-empty output lines + the log path + a '-Diagnose' next-step hint, appended to EVERY fatal error. - Cluster-create failure passes k3d's stdout/stderr so the real reason (image pull / proxy / port / WSL) shows on screen — the motivating case. - Helm repo-add / reconcile / install failures pass helm's output via $Detail instead of embedding it (no more duplicated log-path text). - Install log path is announced up front in the banner (was log-only before). Closes#423 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#423): force array enumeration of Err detail lines (Bugbot, defensive) Bugbot flagged that a single-line Get-ErrDetailLines return (no detail + no LOG_FILE, e.g. a Confirm-Config failure before Start-InstallLog) unwraps to a scalar string. The foreach statement already iterates a scalar once (verified: it prints the whole line, not per-character), so the reported char-splitting does not reproduce -- but wrap the enumeration in @(...) to make that unambiguous and future-proof. Adds a regression test asserting the single-line case stays one intact line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#423): strip PS 5.1 ErrorRecord chrome from the failure excerpt (Bugbot) helm failures arrive as `native 2>&1 | Out-String`; on Windows PowerShell 5.1 that wraps stderr in ErrorRecord chrome (the `At <file>:<n> char:<n>` position line plus the `+ ...` / `+ CategoryInfo` / `+ FullyQualifiedErrorId` block). Get-ErrDetailLines kept only the last 5 non-empty lines, so the excerpt was all chrome and the real `Error:` line dropped out -- a regression from the previous full-message dump. Filter those chrome lines before taking the window so the actual error survives. Adds a regression test simulating the 5.1 rendering. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#423): drop duplicate inline log-path hints (Bugbot) Err now always prints the log path via Get-ErrDetailLines, so the k3d spawn-failure and create-timeout paths that still Hint "Full log:" right before Err printed it twice. Remove those inline hints; Err is the single source. Adds a guard test asserting no inline 'Full log:' hints remain in the installer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#423): put stderr last in the create-failure Err detail (Asad) Get-ErrDetailLines keeps the LAST 5 non-empty lines, so with detail ordered stderr-then-stdout any k3d stdout tail could crowd the real stderr reason (FATA/x509/port) out of the excerpt. Order it stdout-then-stderr so the stderr tail survives the window; also matches the Write-HostCaCreateHint order just above. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Syed Is Saqlain <saqlain.syed007@gmail.com> Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Arturo Peroni <arturo@tracebloc.io> Co-authored-by: shujaat_tracebloc <153823837+shujaatTracebloc@users.noreply.github.com> Co-authored-by: shujaat hasan <shujaat@tracebloc.io> Co-authored-by: tracebloc-release-train[bot] <309815517+tracebloc-release-train[bot]@users.noreply.github.com> * ci(1606): standard-checks and helm-ci call the Makefile instead of copying it CI parity. Two jobs restated targets the Makefile already declares -- and one of them had ALREADY DRIFTED, in the direction nobody notices. LINT. This job spelled out the shellcheck file list inline while the Makefile kept the same list in SHELLCHECK_FILES. Measured: the Makefile carries 19 entries, this file carried 9. Ten scripts were shellchecked on a contributor machine and NOT at the merge gate: gen-manifest.sh check-facts.sh check-style.sh lib/*.sh tests/check-drift.sh tests/e2e-full-seal.sh tests/e2e-journey.sh tests/path-persist.sh tests/chart-env-vocabulary.sh tests/env-vocabulary-agreement.sh gen-manifest.sh is the installer integrity-manifest generator, so a shell defect there could not be caught by this gate. `make lint` is green across all 19 on this tree, so arming the full list imports no backlog. HELM LINT. The values-file loop, `helm lint --strict ./ingestor` and both vocabulary scripts were verbatim copies. `env-vocabulary-agreement.sh` exists to prove the four CLIENT_ENV declarations agree with each other (backend#1729 sweep 5) -- a check about "these declarations must not drift" being itself declared twice is the joke version, and a third vocabulary script added to the Makefile alone would leave this gate silently not running it. The apt install of shellcheck stays: it bootstraps the runner, it is not a duplicated command. Job names untouched. `Lint` and `Unit tests` are required status checks on main, matched by name. Verified: make lint, make helm-lint and make helm-vocab all exit 0 on this tree; actionlint clean on both files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Asad Iqbal (Saadi) <asad.dsoft@gmail.com> Co-authored-by: shujaat hasan <shujaathasan@shujaats-MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: shujaat_tracebloc <153823837+shujaatTracebloc@users.noreply.github.com> Co-authored-by: Arturo Peroni <arturo@tracebloc.io> Co-authored-by: tracebloc-release-train[bot] <309815517+tracebloc-release-train[bot]@users.noreply.github.com> Co-authored-by: Syed Is Saqlain <saqlain.syed007@gmail.com> Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local> Co-authored-by: shujaat hasan <shujaat@tracebloc.io>
LukasWodka added a commit
that referenced
this pull request
Aug 21, 2026
…r, namespace gate Bugbot's second round on #779. All three verified against the rendered chart before fixing. 1. HIGH — `optional: true` DOES NOT MEAN "buffer, don't fail". It stops the kubelet failing the MOUNT; it does not make the file appear, and `bearertokenauth` needs `filename` to resolve. Whether a missing file aborts the extension's Start is version-dependent and lives in core's `credentialsfile.ValueResolver`, not the extension — I could not establish it for 0.159.0 from that tag or from `main`. So this does not BET on the answer. A `lookup`-guarded pre-flight (the same idiom as resource-monitor's metrics-server probe: empty during `helm template`, so offline rendering is unblocked) refuses the RELEASE with a message naming the missing Secret and namespace. Either upstream behaviour is then fine, because the case is unreachable — which beats reading the source correctly and depending on it. It is also the right direction for a fail-soft component: "the install said no, here is the Secret it wants" is actionable in a way that CrashLoopBackOff on 200 nodes is not. 2. MEDIUM — `global.imageRegistry` was ignored. Squid and the other third-party images use #585 precedence: global mirror, then per-image registry, then docker.io. Passing `image.registry` alone meant a mirrored or air-gapped fleet would ImagePullBackOff the Collector while everything else pulled fine. Both ends of the chain are now asserted — a precedence chain tested at one end only is half-tested. 3. MEDIUM — the Collector is a SECOND TENANT of the node-agents namespace, and the gates had not caught up. `node-agents-namespace.yaml` and the mirrored pull Secret in `docker-registry-secret.yaml` were still gated on `resourceMonitor != false`, so `resourceMonitor: false` + `telemetryCollector.enabled: true` produced a DaemonSet targeting a namespace this chart never created, with no pull Secret. Every dependent resource now shares the feature's gate. Proven both ways: with the fix, 4 resources land in that namespace; reverting the pull-Secret gate drops it to 3. FOUND WHILE FIXING, AND MINE: `telemetryCollector` was the ONLY chart-owned top-level key of 36 absent from `values.schema.json` (`global` is a Helm built-in). So the whole block was unvalidated — a typo in `classAContainers` would have been silently accepted and the Collector would have collected nothing, which is finding #1 of the first round arriving by a different route. Added, with `authScheme` as a CLOSED enum so a typo cannot pick a third value, and `type: [object, null]` so `--reuse-values` from a release predating the block still validates. Tests: 25 in the Collector suite. The three fixes are mutation-proved, including a vacuity check — widening the namespace gate to `true` also reddens, so the positive test is not passing for free. Chart 1.9.54. Whole suite 516/516 across 34 suites; `make drift` green on 7 guards; lint clean; manifest current. A process note for my own future reference: three of my debugging renders came back empty and I briefly read that as the fix not working. All three were invalid test inputs the schema was correctly rejecting — with `2>/dev/null` hiding the reason. Don't suppress stderr while diagnosing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
7 tasks
LukasWodka added a commit
that referenced
this pull request
Aug 21, 2026
…ckend#1906) (#779) * feat(telemetry): the edge Collector, Class A only, shipping inert (backend#1906) RFC-BACKEND-1872 D6/D7. An OpenTelemetry Collector DaemonSet that reads Class A container stdout with `filelog` and forwards to the backend's ingest endpoint. SHIPS DISABLED, and that is the load-bearing decision rather than caution. The exporter authenticates with a token this chart does not create: jobs-manager writes it into a Secret and does not do so yet. Enabling it first would put a DaemonSet on every customer node spooling to disk with nothing it can deliver -- filling toward the 1 GiB cap per node for no benefit, which is exactly the "telemetry must never be the reason a node fills" risk D7 exists to bound. Same posture as `egressProxy.routeWorkloads`. A DAEMONSET, because the kubelet writes container stdout per NODE. A Deployment would see only the node it landed on and report healthy while collecting a fraction of the fleet -- the shape of silence this epic exists to remove. CLASS A ONLY (D12). The include globs name the four control-plane containers this chart owns; a bare wildcard would sweep in training and ingestion pods, which are Class B and gated on backend#1908. Today's bounding is SECRET redaction, not CONTENT redaction, and raw customer cell values have already reached central telemetry once (backend#1879), so the narrow scope is the protection. THE REDACTION FLOOR IS RELOCATED, not re-invented -- D6 precondition 2. Reading stdout through `filelog` BYPASSES controller.py's app-side handler, so without it this change would silently REMOVE a protection. All six `_LOG_REDACTIONS` patterns are carried (client-runtime@45006ec), translated Python -> RE2 deliberately: `\1` becomes `$1`, `(?i)`/`(?m)` kept, and no pattern uses lookaround or in-pattern backreferences, which RE2 rejects -- checked, because a silently-invalid regex in OTTL is a processor that starts and scrubs nothing. All six were compiled as RE2 and run against sample secrets; the `SharedAccessKeyName` carve-out controller.py documents survives. THIS IS A CROSS-REPO SECOND COPY THAT CANNOT BE MACHINE-CHECKED FROM HERE, and the comment says so rather than implying coverage: the producer is Python in another repo, Helm cannot import it, and no test in this repo can detect drift. D7's bounds, all three: byte cap 1 GiB via `sizer: bytes`; `max_elapsed_time: 0` so the cap is the ONLY bound; disk-backed `file_storage` on a hostPath so the queue survives the pod restart that is the common case during an outage. `block_on_overflow: false` is drop-NEWEST per D7's 2026-08-20 amendment (rfcs#36) -- `exporterhelper` sheds at the entrance and has no evict-oldest option -- which is why the Collector's own metrics are exposed: the drop count is the only signal separating a quiet edge from a shedding one. `scheme: Token`, NOT the extension's default `Bearer`. The edge holds a DRF user token and the endpoint accepts it via `TokenAuthentication` under `IsAuthenticatedEdge`; `Bearer` is the other credential type and 401s silently. The token is read through `bearertokenauth`'s `filename`, which watches the projected file (credentialsfile.ValueResolver + WithOnChange) -- verified in the extension's source, because the README does not mention reloading and #1906's "a projected update needs no restart" rests on it. `logs_endpoint`, not `endpoint`: otlphttp appends /v1/logs to the latter and the ingest boundary is a versioned path of ours. One defect caught by rendering rather than reading: Helm parses 1073741824 as a float64 and emitted `queue_size: 1.073741824e+09`, a YAML float where the Collector wants an integer -- the cap would not have been the number in values.yaml. `| int64` fixes it and a test asserts no scientific notation. Verified: helm lint clean; the WHOLE chart suite 510/510 across 34 suites; the new suite 19/19; `chart-version-guard.bats` 23/23 after bumping version and appVersion together to 1.9.51 (the guard requires a bump when chart content changes). 15 mutations, all killed -- enabling by default, widening filelog to a wildcard, dropping redaction from the pipeline, losing a redaction pattern, losing `int64`, reverting `sizer`, adding a second time bound, falling back to memory, reverting to `Bearer`, using `endpoint`, mounting host logs writable, making the Secret required, moving the queue to an emptyDir, removing the config checksum, and granting a ClusterRole. NOT in scope: retiring the two App Insights entries from the squid ACL. That cuts client-runtime's live AzureLogHandler path and belongs to #1910's cutover, not here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(telemetry): the Collector gate broke `helm upgrade --reuse-values` Caught by the Fleet auto-upgrade E2E, and it was far worse than the feature not working: it broke the UPGRADE, on every existing customer. Error: UPGRADE FAILED: template: telemetry-collector-rbac.yaml:1:14: <.Values.telemetryCollector.enabled>: nil pointer evaluating interface {}.enabled `helm upgrade --reuse-values` replays the values STORED WITH THE PREVIOUS RELEASE and does not merge values.yaml defaults. Every release made before this block existed therefore has no `telemetryCollector` key at all, so the bare gate is a nil-pointer dereference and the whole release fails. THE CHART ALREADY DOCUMENTS THIS TRAP AND I DID NOT FOLLOW IT. resource-monitor-daemonset.yaml says of `images.resourceMonitor`: "older releases that upgrade via `helm upgrade --reuse-values` won't have the block in their stored values, so read it defensively (nested `default dict` tolerates a missing `images` map AND a missing `resourceMonitor` entry). `dig` is not usable here -- it rejects chartutil.Values." Same idiom now applies here. Every read goes through `{{- $tc := default (dict) .Values.telemetryCollector -}}` with nested maps defaulted the same way, and each defaulted value repeats the values.yaml default so a partial stored map cannot render a half-configured Collector. An old release upgrading this way gets NO Collector, which is right twice over: it matches `enabled: false`, and a feature must never arrive on a cluster via a values map the operator never saw. The regression test sets `telemetryCollector: null` -- ABSENT, not `false`, because absent is the case that broke and false is a key that exists (already covered). Mutation-proved: reverting the gate reproduces the E2E's exact error and the test errors rather than passing. Chart bumped to 1.9.52 (the version guard requires it on chart content change). Whole chart suite 511/511 across 34 suites; the new suite 20/20. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(telemetry): the Collector collected nothing, and ran an image that could not start Two Bugbot Highs on #779, both correct, and both the same class: a Collector that starts, reports healthy, and ships nothing. 1. EVERY FILELOG GLOB MATCHED NOTHING. `classAContainers` listed `tracebloc-jobs-manager`, `egress-proxy` and `requests-proxy` -- WORKLOAD names. `filelog` matches the kubelet's on-disk path, which carries the CONTAINER name, and the real ones are `api` (jobs-manager), `pods-monitor-container` (Dockerfile.controller, i.e. D6's `controller.py`) and `squid`. Worse, every path was scoped to `.Release.Namespace` while resource-monitor is a DaemonSet in `nodeAgents.namespace` -- so a whole Class A component was unreachable even had its name been right. All four globs targeted nothing. `proxy` (requests-proxy) is deliberately NOT added: it runs the jobs-manager image but is not in D6's Class A list, and admitting a container because it shares an image is how a class boundary stops meaning anything. 2. THE IMAGE PREDATED ITS OWN CONFIG. Pinned 0.109.0, while the pipeline sets `block_on_overflow` (upstream 2025-03, ~0.122+) and `sizer: bytes` on a persistent queue (~0.130). The Collector REJECTS unknown configuration keys, so this is not a cap silently unapplied -- it is a container that does not start, on every node. Now 0.159.0. The mistake worth naming: I verified `sizer` and `block_on_overflow` against CURRENT upstream docs and then pinned an image from before they existed. Verifying a feature is not verifying the version that has it. WHAT KEEPS BOTH FIXED. A new derived guard, `scripts/tests/collector-class-a-agreement.sh`, wired into DRIFT_GUARDS (7 now). It holds NO list of names: it renders the chart, reads container names out of every workload, reads the globs out of the Collector's own ConfigMap, and compares. A hand-written expectation would have agreed with whichever side it was copied from -- which is exactly how the original passed review, and why the helm-unittest assertions could not catch it either (they asserted the same wrong names). It is a cross-DOCUMENT agreement, so helm-unittest cannot express it: that plugin asserts within one template at a time. Hence a shell test. It fails closed -- zero globs or zero containers is a finding, since two empty sets compare equal. Mutation-proved against BOTH findings: restoring the old names, and re-scoping node-agents to the release namespace, each reddens it with the offending pair named. The image floor is asserted as a FLOOR in the helm suite, not an exact tag -- an exact pin there would be a second place to update on every bump and would pass by agreeing with itself. shellcheck earned its place too: it caught `render | python3 - <<'PY'`, where the heredoc overrides the pipe so the comparison read an empty document set. It failed CLOSED ("found 0 Collector ConfigMaps") rather than passing, which is the design working, but SC2259 named the cause directly. Chart 1.9.53. Whole chart suite 512/512 across 34 suites; the Collector suite 21/21; `make drift` green on all 7 guards; manifest up to date. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(telemetry): three more on the Collector — token pre-flight, mirror, namespace gate Bugbot's second round on #779. All three verified against the rendered chart before fixing. 1. HIGH — `optional: true` DOES NOT MEAN "buffer, don't fail". It stops the kubelet failing the MOUNT; it does not make the file appear, and `bearertokenauth` needs `filename` to resolve. Whether a missing file aborts the extension's Start is version-dependent and lives in core's `credentialsfile.ValueResolver`, not the extension — I could not establish it for 0.159.0 from that tag or from `main`. So this does not BET on the answer. A `lookup`-guarded pre-flight (the same idiom as resource-monitor's metrics-server probe: empty during `helm template`, so offline rendering is unblocked) refuses the RELEASE with a message naming the missing Secret and namespace. Either upstream behaviour is then fine, because the case is unreachable — which beats reading the source correctly and depending on it. It is also the right direction for a fail-soft component: "the install said no, here is the Secret it wants" is actionable in a way that CrashLoopBackOff on 200 nodes is not. 2. MEDIUM — `global.imageRegistry` was ignored. Squid and the other third-party images use #585 precedence: global mirror, then per-image registry, then docker.io. Passing `image.registry` alone meant a mirrored or air-gapped fleet would ImagePullBackOff the Collector while everything else pulled fine. Both ends of the chain are now asserted — a precedence chain tested at one end only is half-tested. 3. MEDIUM — the Collector is a SECOND TENANT of the node-agents namespace, and the gates had not caught up. `node-agents-namespace.yaml` and the mirrored pull Secret in `docker-registry-secret.yaml` were still gated on `resourceMonitor != false`, so `resourceMonitor: false` + `telemetryCollector.enabled: true` produced a DaemonSet targeting a namespace this chart never created, with no pull Secret. Every dependent resource now shares the feature's gate. Proven both ways: with the fix, 4 resources land in that namespace; reverting the pull-Secret gate drops it to 3. FOUND WHILE FIXING, AND MINE: `telemetryCollector` was the ONLY chart-owned top-level key of 36 absent from `values.schema.json` (`global` is a Helm built-in). So the whole block was unvalidated — a typo in `classAContainers` would have been silently accepted and the Collector would have collected nothing, which is finding #1 of the first round arriving by a different route. Added, with `authScheme` as a CLOSED enum so a typo cannot pick a third value, and `type: [object, null]` so `--reuse-values` from a release predating the block still validates. Tests: 25 in the Collector suite. The three fixes are mutation-proved, including a vacuity check — widening the namespace gate to `true` also reddens, so the positive test is not passing for free. Chart 1.9.54. Whole suite 516/516 across 34 suites; `make drift` green on 7 guards; lint clean; manifest current. A process note for my own future reference: three of my debugging renders came back empty and I briefly read that as the fix not working. All three were invalid test inputs the schema was correctly rejecting — with `2>/dev/null` hiding the reason. Don't suppress stderr while diagnosing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(telemetry): pin the pull-secret gate, and that the pre-flight stays offline-safe @aptracebloc's two non-blocking follow-ups on #779. Taken now rather than deferred because the conflict-resolution push dismissed the approval anyway, so they cost no extra review round. 1. THE PULL-SECRET GATE WAS NOT MUTATION-PINNED while the namespace gate was. I had verified it by hand — render diff, 4 node-agents resources with the fix and 3 without — and a by-hand check leaves nothing behind. That was the gap, not the fix. Now pinned in BOTH directions: narrowing the gate back reddens the positive test, and widening it to `true` reddens the negative one, so neither passes for free. It needs real registry values, which is worth recording: `dockerRegistry.server` is `format: uri` and `email` is required by its `allOf`, so a partial set is rejected by values.schema.json before any template renders — and helm-unittest surfaces that as a plugin ERROR rather than a template failure, which reads exactly like a code defect. It cost me three confused renders earlier, all of them my own invalid inputs with stderr suppressed. 2. THE PRE-FLIGHT `fail` IS UN-PINNABLE and he is right that it is: `lookup` returns empty under `helm template`, so the guard never fires in a unit test. But the INVERSE is testable and is the failure that would actually hurt — a pre-flight that fired offline would break every `helm template`, every CI render and every `--dry-run` in the fleet. That property is now asserted directly instead of resting on the guard's own comment. Chart 1.9.53. Whole chart suite 519/519 across 34 suites; the Collector suite 28/28; `make drift` green on all 8 guards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(telemetry): select root for the Collector, and give it an OpenShift SCC (backend#1906) Both of Bugbot's findings on #779. Both verified against upstream before fixing — they are real, and the second one is the more dangerous of the two. 1. THE DISK-BACKED QUEUE WAS UNWRITABLE. `runAsNonRoot: false` PERMITS root; it does not SELECT it. The pinned otelcol-contrib image declares `USER 10001` (`ARG USER_UID=10001` in the upstream Dockerfile), and the kubelet creates a DirectoryOrCreate hostPath root-owned 0755 — so the Collector ran as 10001 and could not write /var/lib/tracebloc/<release>/telemetry. D7's whole point is a queue that survives a restart; it would have failed the moment the Collector was enabled, which is exactly the kind of defect that hides behind a chart that renders clean. Fixed by pinning `runAsUser: 0`. Root rather than an init-container chown for a reason beyond one fewer container: the READ side needs it too — container log files under /var/log/pods are root-owned and not world-readable on every runtime, so a non-root collector may not be able to read the logs it exists to read. fluent-bit and the cloudwatch-agent already on these clusters run as root for the same reason. It is a NARROW root, and that is the trade being made rather than glossed: all capabilities dropped, no privilege escalation, read-only root filesystem, and the only writable paths are the queue and /tmp. 2. ON OPENSHIFT THE COLLECTOR HAD NO SCC AT ALL. Chart-managed SCCs gated on `resourceMonitor` alone and bound only that ServiceAccount, so a second hostPath DaemonSet with its own SA was refused at admission. A SEPARATE SCC, not a widening: resource-monitor's declares MustRunAsNonRoot, which is correct for it — read-only mounts, no write anywhere — and loosening it to RunAsAny to serve the Collector would hand a broader run-as rule to a workload that does not need one. WHY A NEW DRIFT GUARD. This is the second cross-document gap on this PR to get through a green 519-test suite (the first was the Class A globs), because the defect is the ABSENCE of a relationship between two documents and helm-unittest asserts within one template at a time. scripts/tests/openshift-scc-coverage.sh renders the chart with OpenShift on and compares the hostPath workloads against the SCCs and their `users:` lists — holding no list of its own, deriving both sides, failing closed when either side is empty. It checks run-as compatibility too, and that is the point rather than a flourish: adding the Collector's SA to resource-monitor's existing SCC would satisfy "is it covered" while still failing at admission, since that SCC refuses root. A guard that checked only membership would have gone green on the wrong fix. MUTATION-PROVEN, anchors asserted applied in every case: guard M1 run-as -> MustRunAsNonRoot KILLED (root refused by ...) M2 allowHostDirVolumePlugin off KILLED (no SCC) M3 users emptied KILLED (no SCC) M4 template deleted KILLED — reproduces the original defect unittests M1 drop `runAsUser: 0` 1 failed <- the shipped regression M2 SCC -> MustRunAsNonRoot 1 failed M3 gate on openshift only 2 failed M4 gate on Collector only 1 failed M5 users emptied 1 failed No survivors; green again on restore. 525/525 chart tests, 9/9 drift guards, shellcheck -S warning clean. `scripts/manifest.sha256` deliberately NOT regenerated: it covers only the sub-scripts install.sh fetches and hash-verifies, and a CI guard under scripts/tests/ is not one — checked rather than assumed, after #775. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(telemetry): the filelog receiver kept its read offsets in memory (backend#1906) Bugbot's find, and correct. `file_storage` backed only the exporter queue. The `filelog` receiver set no `storage`, so its read offsets lived in memory — and `start_at` governs where a NEWLY DISCOVERED file is read from, not where a known one resumes. With nothing persisted, every file looks new after a restart, `end` wins, and everything written while the Collector was down is skipped. Silently: no error, no gap in any metric. D7's disk queue covers records already ingested. Nothing covered the read side, and restart-during-a-backend-outage is precisely the path that needs both. THE COMMENT THAT SAT THERE ASSERTED THE OPPOSITE — that `start_at: end` was what stopped re-reads across restarts. It is why this was not caught in review: a wrong comment is worse than none, because it answers the question before anyone asks it (backend#1729 rule 7). Corrected rather than deleted, since the reason `end` is still right needs saying: with offsets persisted it applies only to files with no stored position, so a fresh install does not ingest the whole existing backlog while a known file resumes exactly where it stopped. The receiver shares the exporter's extension. `file_storage` keys entries by component so they cannot collide, and offsets cost kilobytes against the queue's cap. It writes to the same hostPath — which is only writable because the previous commit pinned `runAsUser: 0`, so the two findings are more connected than they looked. WHY A GUARD AND NOT A helm-unittest ASSERTION. The Collector's config is a YAML document embedded in a string inside the ConfigMap. helm-unittest can only regex that string, and a regex for `storage: file_storage` matches the EXPORTER's queue setting just as happily as the receiver's — passing while the receiver has none, which is the exact defect. Parsing is the only way to say where the key is, so scripts/tests/collector-offsets-persisted.sh parses it and checks the named extension is both declared and enabled in `service.extensions`: a storage extension that is configured but not switched on is silently ignored. Mutation-proven, anchors asserted applied: filelog `storage` removed KILLED — reproduces the original defect points at an undeclared extension KILLED declared but absent from service.exts KILLED No survivors; green on restore. 525/525 chart tests, 10/10 drift guards, shellcheck -S warning clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(telemetry): one predicate for the node-agents namespace, not five copies (backend#1906) @saadqbal's review of #779, and he is right that Bugbot's High was one instance of a class. Five templates put something in `nodeAgents.namespace` and each held its own copy of "is resource-monitor on"; this PR widened two of them for the Collector and left the rest, so `resourceMonitor: false` + `telemetryCollector.enabled: true` — the configuration the Collector exists to enable — created the namespace, landed the DaemonSet, and left the RBAC that manages it behind. `tracebloc.nodeAgentsInUse` is now the single predicate, and all five call sites read it. The nil-guard lives there too rather than five times over. I VERIFIED EACH INSTANCE RATHER THAN TAKING THE COUNT, and the count was wrong in both directions: * THERE IS A FOURTH the review did not list — `rbac.yaml:183`, jobs-manager's node-agents Role — found by diffing the two renders rather than by reading gates. * TWO OF THE THREE MUST NOT BE WIDENED, and widening them on request would have made this worse: - `secrets.yaml:112` mirrors CLIENT_ID/CLIENT_PASSWORD there so the resource-monitor DaemonSet can read them via secretKeyRef. The Collector authenticates with its own telemetry token and never reads them, so widening this would copy CUSTOMER CREDENTIALS into a namespace for a workload that has no use for them. Left gated on resource-monitor. - `rbac.yaml:183` grants daemonsets get/list/watch so jobs-manager can read resource-monitor's version for the heartbeat inventory. Read-only, about a specific workload, and nothing asks it for the Collector's version — so it is correctly absent. backend#2274 WILL need it, when jobs-manager starts writing the Collector's token Secret there; that is a Secret write, and it belongs to that ticket's chart half. So two gates widened (auto-upgrade, image-refresh — both mutate DaemonSets in that namespace, and the Collector is one), two deliberately not. GUARDED ONCE TOO, because "fix it once" is only half of it. Three careful readings produced three different counts, which is the argument for a machine check. scripts/tests/node-agents-tenancy.sh renders the chart twice and asserts that the Roles which MUTATE DaemonSets in that namespace do not depend on WHICH DaemonSet is there. Deliberately narrower than "the two renders must match": the mirrored Secret and the read-only jobs-manager Role SHOULD differ, and a stricter check would have to be wrong about them. Writing it caught a defect in itself worth recording: matching the literals `apps`/`daemonsets` missed auto-upgrade's Role, which is `apiGroups: ["*"], resources: ["*"], verbs: ["*"]` — the very Role Bugbot flagged. A guard that cannot see the finding it was written for is worse than none. Wildcards count now. Mutation-proven, anchors asserted applied: auto-upgrade gate reverted to resourceMonitor guard KILLED + 1 unittest failed image-refresh gate reverted guard KILLED the predicate hardcoded to `true` (over-gating) 7 unittests failed the nil-guard removed from the predicate 1 failed, 1 errored That last row SURVIVED at first and the reason matters: Go's `or` short-circuits, so my null test set `resourceMonitor: true`, the second operand was never evaluated, and the test passed happily with the guard DELETED. It sets `resourceMonitor: false` now — which is the whole test. Without the mutation it would have shipped asserting nothing, which is this epic's own dominant defect class appearing in the fix for it. The over-gating direction is covered by helm-unittest's negatives rather than by the guard, and that division is stated in both places instead of left implicit. 529/529 chart tests, 11/11 drift guards. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(telemetry): one edge's Collector could ingest another edge's logs (backend#1906) Bugbot's find, and it is a cross-tenant leak rather than a noisy metric. The kubelet's log directory is `<ns>_<podName>_<uid>`. A bare pod wildcard is fine in the release namespace — that namespace belongs to one release — but `nodeAgents.namespace` is SHAREABLE ON PURPOSE, which is exactly why the workloads in it are release-scoped. Container names are not: resource-monitor's container is `tracebloc-resource-monitor` in every release. So the node-agents glob matched EVERY release's resource-monitor in a shared namespace, and one edge's Collector would have read another edge's Class A logs and shipped them under its own ingest token. Both Collectors look healthy throughout. The pod portion is now scoped to the release. DaemonSet pods are `<daemonsetName>-<hash>` and the DaemonSet name is release-scoped, so the release prefix is what separates them. THE RESIDUAL IS STATED, NOT PAPERED OVER. The wildcard crosses `-`, so a release named `edge` and one named `edge-2` sharing this namespace would still cross-match. Closing that needs the workload name rather than the release name, and this list cannot give it: it is a list of CONTAINER names, and container-to-workload is not 1:1 in general. Two releases whose names are prefixes of one another, in one shared namespace, is what remains; the ordinary case is closed. GUARDED, and the guard now checks the property rather than the shape. Updating collector-class-a-agreement.sh's parse for the new glob would have been enough to make it pass, which is the trap — so it asserts the invariant instead: a glob into a namespace that is NOT the release namespace may not use a bare pod wildcard. Both sides derived from the render, including which namespace counts as "the release namespace", read off the Deployments rather than written down. Mutation-proven, anchor asserted applied: reverting the glob to a bare pod wildcard → guard KILLED, naming the namespace and the leak. The unit test pins the release prefix too, so dropping it reddens both tiers. A Go template comment ends at the first `star slash`, so spelling the glob out inline inside the explanatory comment truncated it and broke the render — one debug cycle, and worth the note in the file since the next person to document a glob there will hit it. 529/529 chart tests, 11/11 drift guards. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(telemetry): two more ways the Collector could run healthy and blind (backend#1906) Both Bugbot's, both real, and both the same shape as everything else on this PR: the DaemonSet is Ready, the metrics look quiet, and no records exist. 1. `DirectoryOrCreate` ON A PATH WE DO NOT OWN. `/var/log/pods` is the KUBELET'S directory — we read it. With `DirectoryOrCreate`, a wrong or missing `hostLogsPath` made the kubelet helpfully create an empty directory, and the Collector then started, reported healthy, and matched no files forever. That is the Class A glob guard's failure mode arriving by a different route, and the guard cannot see it: the globs are correct, the directory is just empty. `Directory` fails the pod instead, which is the right direction for a path we expect to already exist — resource-monitor's /proc and /sys have always used it. THE QUEUE KEEPS `DirectoryOrCreate`, and the asymmetry is the design rather than an oversight: that path is ours and does not exist on a node's first install, so `Directory` there would refuse to start the Collector on every fresh node. Both are asserted in one test, because a well-meaning "make these consistent" edit breaks exactly one of the two and the tests should say which. 2. `seLinuxContext: MustRunAs` CANNOT READ CONTAINER LOGS. I copied that from resource-monitor's SCC, where it is correct — that workload reads /proc and /sys. Container logs under /var/log/pods are labelled `container_log_t` with per-container MCS categories, and a namespace MCS context cannot read them. So on OpenShift the pods would be admitted, run as root, and take a permission error on every include glob. `RunAsAny`, the same widening already applied to `runAsUser` on this SCC and for the same reason — resource-monitor's own SCC is untouched and keeps both tighter policies. That copy-from-the-sibling error is worth naming, because it is the second time on this PR: the sibling SCC is the right template to start from and the wrong one to finish with, since every field on it was chosen for a workload that only reads kernel pseudo-filesystems read-only. Mutation-proven, anchors asserted applied: host-logs back to DirectoryOrCreate 1 failed queue changed to Directory 1 failed SELinux back to MustRunAs 1 failed 531/531 chart tests, 11/11 drift guards. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
changes in helm chart