feat(installer): stop-and-check gate — recognize an existing install

[LukasWodka]

Summary

Adds a read-only "stop-and-check" gate to the installer so re-running it on an already-set-up machine no longer drags the user through full provisioning. On a verifiably healthy machine it prints one line and hands off to tracebloc (the home screen), then exits 0. Otherwise it falls through to today's 5-step flow, unchanged.

This is PR 2 of 2 (PR 1 = the status-aware tracebloc home screen, tracebloc/cli#244). Together they fix the "I just wanted to use the CLI / reinstall, and suddenly a whole new environment gets provisioned" problem.

Behavior

The gate runs after the banner, before the 5-step roadmap. assess_existing_install() classifies the machine (read-only, bounded, never fatal):

• healthy — cluster running AND a tracebloc release present AND all core workloads Ready (mysql-client + jobs-manager + requests-proxy) AND the CLI present → prints ✓ Already set up on this machine — no need to run the installer again., then runs tracebloc (the home screen) and exit 0. Never re-provisions, never re-prompts login.
• degraded — cluster stopped, a workload not Ready, or the CLI missing → prints an honest one-liner naming what's off ("your secure environment is stopped / still starting up / the CLI isn't installed"), then falls through to today's flow to reconcile.
• fresh — no cluster, or no tracebloc release → today's 5-step install, unchanged.

--force / --reinstall (or TRACEBLOC_FORCE_REINSTALL=1) bypasses the gate entirely. Non-interactive (curl | bash): healthy still short-circuits (the hand-off is output-only, reads no stdin); degraded/fresh run the normal flow.

Safety (this is a public, cosign-verified installer)

• assess is strictly read-only — k3d cluster list, kubectl get, helm list (via the reused detect_installed_client), has. It never starts the cluster, runs helm upgrade, mints credentials, or writes anything, and it does not call the mutating _handle_existing_cluster.
• No false "healthy." Every probe degrades to non-healthy on any error / timeout / missing tool / uncertainty — a machine that needs setup is never told "already set up." (Mutation-proven.)
• Manifest integrity.scripts/lib/assess.sh is added to both bootstrap FILES arrays and manifest.sha256 is regenerated (gen-manifest.sh --check → up to date), so the cosign bootstrap still verifies; install-bootstrap.bats file-count lists updated (15 → 16).
• The healthy path uses exit 0 (not exec) so the install_cleanup EXIT trap still runs — a verified no-op at gate time (before any credential / sudo-keepalive is set).
• provision.sh untouched, to stay clear of the active release-train: staging -> main #838 provision rewrite.

Single source of truth for "ready"

Extracted the client-workload deployment list into _client_workload_deployments() (common.sh); both wait_for_client_ready (the post-install gate) and the assess readiness probe consume it, so the installer's definition of "ready" and the gate's can't drift.

Code review

Adversarial review before opening — both invariants (no false-healthy; non-mutating + manifest-consistent) confirmed and mutation-proven. Fixed in the second commit:

• [should-fix] the readiness gate checked only jobs-manager; a machine with jobs-manager up but requests-proxy (training egress) or mysql-client down could be called "healthy" → now requires all three, via the shared helper.
• [should-fix] an unguarded awk-pipe assignment that could abort the installer under set -e/pipefail (SIGPIPE) → guarded to match its jq sibling.

Deferred (fast-follow)

Surgical per-layer repair of a degraded machine (start-only / CLI-only / reconcile-only, and skipping re-login when already authenticated) — deliberately left out to keep this PR off provision.sh and clear of #838. Rebases cleanly once #838 lands.

Test plan

• scripts/tests/assess.bats (31 tests): each probe (cluster-running jq + awk fallback; workload-ready per-deployment incl. requests-proxy / mysql-client down; CLI present); all classify branches; the hand-off (runs tracebloc, exit 0, fallback); and the main() branch (healthy → hand-off + exit 0; --force bypass; degraded/fresh → flow). Assertions are fail-loud (grep-backed + single-bracket status) to dodge the macOS bats [[ ]] blindspot; the healthy short-circuit and the all-three-workloads gate are mutation-verified.
• shellcheck --severity=error clean; bash -n clean; gen-manifest.sh --check up to date.
• Full suite: only 2 pre-existing macOS-local failures (reproduce on pristine develop, unrelated files) — Linux CI is authoritative for bats.

Checklist

• Targets develop
• Read-only assess; no false-healthy (mutation-proven)
• Manifest regenerated + bootstrap FILES in sync
• provision.sh untouched (release-train: staging -> main #838 coordination)
• Tests added + mutation-checked
• Human review (Bugbot + reviewer on Monday)

---

Pairs with tracebloc/cli#244. Hand-off degrades gracefully: shows today's home screen on the released CLI, the new status-aware one once cli#244 ships.

---

Note

Medium Risk
Changes behavior of the cosign-verified public installer entrypoint; a false healthy would skip needed setup, though the design biases toward fall-through on any doubt and adds extensive bats coverage.

Overview
Adds a read-only stop-and-check gate so re-running the installer on an already-healthy machine skips full provisioning and hands off to tracebloc instead.

New scripts/lib/assess.sh classifies the host as fresh, degraded, or healthy using bounded k3d/kubectl/helm probes (never mutating). Only healthy short-circuits: running cluster, tracebloc release, all core workloads ready, and CLI present. Uncertainty always falls through to the normal install flow—never a false “already set up.”

install-k8s.sh invokes the gate after the banner; --force / --reinstall and TRACEBLOC_FORCE_REINSTALL=1 bypass it. assess.sh is guarded when missing (stale bootstrap) like other optional libs.

_client_workload_deployments() in common.sh centralizes readiness (mysql-client, jobs-manager, requests-proxy); wait_for_client_ready and the assess gate both use it so “ready” cannot drift.

Bootstrap FILES, gen-manifest.sh, manifest.sha256, help text, and assess.bats / bootstrap tests updated for the new module.

^{Reviewed by Cursor Bugbot for commit 18b113a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

👋 Heads-up — Code review queue is at 46 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• backend#1040 — docs(rfc): configurable preprocessing & imputation (RFC 0003) · author: @LukasWodka · reviewer: @saadqbal
• backend#1076 — fix(taxonomy): drop dead instance_segmentation category (#1005) · author: @LukasWodka · reviewer: @saqlainsyed007
• backend#1077 — feat(global_meta): audit/purge command for legacy per-row GlobalMeta (#1007) · author: @LukasWodka · no reviewer assigned
• backend#1084 — fix(metrics): register semseg metrics — pattern-allow iou_class_N + seed static rows (fix(#1143): add readiness/liveness/startup probes to the requests-proxy Deployment #641) · author: @LukasWodka · reviewer: @saqlainsyed007
• backend#1085 — fix(api): CycleView returns the JSONField confusion matrix; legacy fallback logged (#1075) · author: @LukasWodka · reviewer: @saqlainsyed007
• cli#229 — fix(data): deprecation notices on renamed aliases + clear no-client message (fix(make): drop xargs -r so make lint runs on BSD xargs (backend#2200) #879) · author: @LukasWodka · reviewer: @saadqbal
• cli#230 — fix(data delete): resolve the table name case-insensitively; fail closed (#1027) · author: @LukasWodka · reviewer: @saqlainsyed007
• cli#231 — fix(doctor): surface container restart history, not just crash-loop-now (#1028 item 4) · author: @LukasWodka · no reviewer assigned
• cli#232 — feat(ingest): guide users from dataset to creating a use case · author: @LukasWodka · reviewer: @saadqbal
• cli#233 — fix(data ingest): preflight & message polish — schema-type validation, friendly missing-label, help copy, clearer errors (fix(install): visible wait on held apt lock instead of a silent spinner (#740) #213test: cross-repo fresh-shell last-mile E2E harness (install -> CLI -> cluster info -> push) #214Post-install self-verification of CLI usability (#738) #215fix(client): allow training pods to reach mysql-client (v1.2.1) #76) · author: @LukasWodka · no reviewer assigned

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

[aptracebloc]

Verdict: Approve with nits — the gate is genuinely read-only, biases toward fall-through on any doubt, and is well tested; the only substantive item is a cheap defensive hardening of the healthy hand-off's stdin.

cc @LukasWodka@saadqbal

I reviewed scripts/lib/assess.sh, the main() wiring in install-k8s.sh, the shared _client_workload_deployments helper, and assess.bats, against the surrounding code (detect_installed_client, _cluster_exists, install_cleanup, the bootstrap install.sh). The core safety invariants hold: assess never mutates, never calls the mutating _handle_existing_cluster, has no delete/uninstall path, and every probe degrades to non-healthy on error/timeout/uncertainty. healthy is the only short-circuit and requires all four signals. Confirmed the exit 0 hand-off is safe because at gate time (after print_banner, before Step 3) install_cleanup is a no-op — SUDO_KEEPALIVE_PID and _PROVISION_CRED_FILE are both unset, so its two guarded branches don't fire.

🟡 Minor

• assess.sh:253 — healthy hand-off runs tracebloc with the inherited curl-pipe stdin. Under curl … | bash, install.sh:370 runs bash install-k8s.sh "$@", so the child (and therefore the bare tracebloc in _assess_handoff) inherits the pipe as stdin. || true protects against a non-zero exit but not against a blocking read. The claim "the hand-off reads no stdin" is a property of cli#244, not of the currently-released CLI this path also runs (per the PR's own "degrades gracefully to today's home screen"). Cheap, CLI-independent hardening: tracebloc </dev/null || true. Then a healthy curl | bash re-run can never hang regardless of what bare tracebloc does on any CLI version.

• install-k8s.sh:48-53 — the --force/--reinstall → TB_FORCE_REINSTALL arg translation in main() is untested.assess.bats exercises the bypass by pre-setting TB_FORCE_REINSTALL=1 directly; nothing covers the argv scan that actually sets it. A rename/typo there (the one path a user hits with ./install-k8s.sh --force) would pass CI. The gate logic itself is well covered — this is the one wiring seam left unguarded. A small test sourcing the loop, or asserting --reinstall is honored, would close it.

⚪ Nits

• assess.sh:236 "healthy" release signal can in principle match a non-tracebloc release.detect_installed_client treats any Helm release whose CHART column matches client-[0-9]and whose values contain a clientId as "ours". The clientId requirement makes a real-world collision very unlikely, and this is pre-existing shared logic (reused deliberately for consistency with the Installer mints a new client before the one-client-per-machine guard — orphans it on the backend #303 ownership guard), so no change requested — noting it only because the gate now also keys "already set up" off it.

• assess.sh header says the module "writes nothing". Accurate about install/cluster state, but detect_installed_client (invoked from _assess_classify) does helm get values into a local temp file it owns and rms. Harmless and read-only w.r.t. the machine; the header wording is just slightly stronger than literal.

What's good

• Read-only + fail-safe by construction. Every probe (_assess_cluster_servers_running, _assess_workload_ready, _assess_cli_present) returns non-healthy on any error, missing tool, or non-numeric output; _assess_classify requires positive certainty at each layer before reaching healthy. A machine that needs setup is never told "already set up".
• The shell-robustness class of bug is handled correctly. The awk-pipe SIGPIPE guard (… | awk … } ' ) || line="", assess.sh:145) mirrors its jq sibling; the kubectl … || return 1 capture and the k3d … || running="0" capture all keep set -euo pipefail from aborting the installer on a subshell non-zero. The if [[ -f assess.sh ]]; then source; fi guard (not && source) and the declare -F assess_existing_install check in main() both correctly avoid tripping set -e on a stale bootstrap.
• Single source of truth for "ready". Extracting _client_workload_deployments so both wait_for_client_ready and the assess probe consume it genuinely prevents drift, and requiring all three workloads (not just jobs-manager) is the right fix — the per-workload-down bats cases are mutation-real against the old probe.
• exit 0 (not exec) preserving the EXIT trap is the correct call, and I verified the trap is a no-op at gate time.
• Test coverage is thorough: every classify branch, both cluster-probe paths, per-workload readiness, CLI-in-~/.local/bin, the hand-off (incl. bare-invocation and unresolvable-CLI fallback), the --force bypass, and mutation guards on the short-circuit. Manifest + bootstrap FILES + file-count all updated in lockstep.

🤖 Generated with Claude Code

[aptracebloc]

@LukasWodka — Approved. The core invariants hold: the gate is read-only, every probe fails safe toward the normal flow, the set -euo pipefail patterns are correct, and the exit 0 hand-off is safe (detail in my comment above).

Caveats — follow-up, none blocking:

• 🟡 the healthy hand-off runs bare tracebloc on the inherited curl-pipe stdin; || true guards a non-zero exit but not a blocking read — tracebloc </dev/null || true is a cheap hardening if you're already in the file, otherwise fine as a follow-up.
• 🟡 the --force/--reinstall → TB_FORCE_REINSTALL arg loop is untested (bats presets the var directly).
• ⚪ "release present" could in principle match a non-tracebloc client-* helm release; header "writes nothing" is slightly stronger than literal (detect_installed_client writes a temp file).

This sits squarely in the R8 trust root (install.sh, install-k8s.sh, gen-manifest.sh, lib/*), so Asad's review is the intended backstop — but the gate isn't enforced, so you're clear to merge on my approval.

🤖 Generated with Claude Code