fix(installer): tracebloc upgrade updates a healthy-but-behind CLI (backend#2253)

[aptracebloc]

Problem

tracebloc upgrade can never clear the CLI's update nag (backend#2253). The nag fires when current < LATEST, but the installer's stop-and-check gate hands a verifiably-healthy machine straight to the home screen — "already set up — no need to run the installer again" — and updates nothing. Its CLI floor stops at 0.10.0 (below = mandatory reinstall), so a CLI at e.g. 0.10.5 with latest 0.10.8 clears the floor, classifies healthy, and is nagged on every command while the very command the nag names changes nothing. Two unrelated definitions of "up to date".

Raising MIN_SUPPORTED_CLI_VERSION (backend#1920) was rejected: any floor below latest re-opens the gap at the next release, and a floor pinned to latest makes every release a forced reinstall.

Fix (issue's preferred option 1)

Bridge the two definitions without weakening the floor:

• assess.sh — new read-only _assess_cli_behind_latest, gated on TB_UPGRADE_CLI, compares the installed CLI against TB_CLI_LATEST (resolved and passed by the CLI — no network in the classifier). _assess_classify emits a distinct cli-behind-latest reason, ordered after the floor check so a below-floor CLI stays cli-outdated (still a mandatory full reinstall). Inert on every ordinary installer run.
• install-k8s.shmain() — on cli-behind-latest, update only the CLI (a small, isolated download via upgrade_cli_only) and exit — no full reconcile. assess.sh stays a read-only classifier; the CLI-install mutation lives in main() next to create_cluster/install_client_helm, honoring the gate's non-mutating contract.
• install.sh — TB_UPGRADE_CLI=1 skips the bootstrap's healthy bailout so the run reaches the gate, without forcing a reinstall (_tb_force stays 0).

The CLI half — tracebloc upgrade setting TB_UPGRADE_CLI / TB_CLI_LATEST — is in tracebloc/cli (sibling PR, links below). Windows is unaffected: tracebloc upgrade there runs the CLI's own CLI-only install.ps1, which has no healthy fast-path to skip.

Acceptance

• ✅ Healthy machine, CLI behind latest → cli-behind-latest → CLI updated to latest → nag gone next command.
• ✅ Below-floor still forces a full reinstall (cli-outdated, floor unchanged).
• ✅ New failing-today test: _assess_classify: upgrade intent + CLI behind latest -> degraded (cli-behind-latest) (before this, it classified healthy and nothing ran).

Tests

New bats coverage in assess.bats (behind-latest probe: intent-gating, 2-vs-10 numeric compare, fail-safe-to-update on unknown latest, floor-wins ordering, the early-return), install-cli.bats (upgrade_cli_only runs the CLI step, reported version == latest, stale-bootstrap guard), and install-bootstrap.bats (TB_UPGRADE_CLI skips the bailout, propagates the flag, does not force a reinstall). manifest.sha256 regenerated. check-style.sh clean.

Closes tracebloc/backend#2253

— drafted with Claude Code

---

Note

Low Risk
Changes are gated on explicit TB_UPGRADE_CLI; ordinary installer runs and the CLI floor semantics stay the same, with broad bats coverage and no weakening of bootstrap verification.

Overview
Fixes backend#2253: on a fully healthy machine, tracebloc upgrade used to hit the stop-and-check gate, classify healthy, and do nothing—while the CLI’s update nag compares against latest, not the installer’s 0.10.0 floor.

assess.sh adds _assess_cli_behind_latest, active only when TB_UPGRADE_CLI=1, comparing the installed CLI to TB_CLI_LATEST (no network in the classifier). Classification emits cli-behind-latest after the floor check so below-floor stays cli-outdated. For that reason, assess_existing_install returns without healthy handoff or degraded ceremony.

install.sh sets _tb_bail_ok=0 when TB_UPGRADE_CLI=1 so the bootstrap reaches install-k8s.sh, without exporting TB_FORCE_REINSTALL.

install-k8s.sh calls new upgrade_cli_only when state is cli-behind-latest, then exits—CLI download only, no full reconcile.

install-cli.sh implements upgrade_cli_only: runs install_tracebloc_cli, surfaces the same k3s/GPU advisories as the healthy handoff, and exits non-zero if the CLI is still verifiably behind TB_CLI_LATEST after install.

manifest.sha256 and bats coverage in assess.bats, install-bootstrap.bats, and install-cli.bats are updated accordingly. The CLI setting TB_UPGRADE_CLI / TB_CLI_LATEST is expected from the sibling tracebloc/cli PR.

^{Reviewed by Cursor Bugbot for commit 0f3ce89. Bugbot is set up for automated code reviews on this repo. Configure here.}

[aptracebloc]

Sibling PR (CLI half): tracebloc/cli#590 — tracebloc upgrade sets the TB_UPGRADE_CLI / TB_CLI_LATEST signals this installer reads. Both must land for tracebloc/backend#2253 to close.

— drafted with Claude Code

[LukasWodka]

Early note while CI is still running — I checked the thing a reviewer of either half alone would miss, and it holds.

This and tracebloc/cli#590 are the two ends of one contract for backend#2253: the CLI bakes TB_UPGRADE_CLI=1 into the command and passes the already-resolved TB_CLI_LATEST; this installer has to honour both. A cross-repo pair's characteristic failure is the halves quietly disagreeing — and here a disagreement would mean tracebloc upgrade still does nothing, which is precisely the bug being fixed.

Names agree.TB_UPGRADE_CLI and TB_CLI_LATEST on both sides, and the CLI bakes the flag into the command string rather than the environment, so a copy-pasted manual retry still carries it (upgrade.go:29).

Semantics agree, which is the part worth checking.cli/internal/cli/upgrade.go:79 states the contract as "lacking TB_CLI_LATEST, still upgrades unconditionally under TB_UPGRADE_CLI". scripts/lib/assess.sh:244-247 implements exactly that:

latest="${TB_CLI_LATEST:-}"; latest="${latest#v}"
[[ "$latest"=~ ^[0-9]+(\.[0-9]+)*$ ]] ||return 0 # missing/unparseable → "behind" → update
[[ "$ver"=~ ^[0-9]+(\.[0-9]+)*$ ]] ||return 0
_version_lt "$ver""$latest"

return 0 is "behind latest", so an absent TB_CLI_LATEST updates rather than short-circuits. The comment above it reasons the same way and in the same direction — "Fail SAFE toward updating … 'we cannot prove you are current' -> update, which is exactly what the user asked for" — and only a latest that is both readable and already met returns "not behind". That's the correct direction for this gate: the user typed upgrade, so uncertainty should resolve toward doing it.

Gating the whole classifier on TB_UPGRADE_CLI so it stays inert on ordinary runs is also the right call — it keeps the normal path free of a "what is latest?" lookup rather than making every install pay for this feature.

Mechanical gates, so you can clear all three in one pass (none of these is a code problem):

• this PR — closing-ref: the title names backend#2253 but the body links nothing. Add Closes tracebloc/backend#2253 (full form — a bare Closes #2253 resolves against client, not backend).
• cli#590 — same closing-ref, plus version-bump-gate / version-check.
• client#865 — chart-version-guard.sh: chart content changed without a Chart.yaml version bump.

I'll do the substantive review of both halves once they're green — this note is only the cross-repo contract, checked early because it's the one thing that gets missed when two PRs are reviewed separately.

[LukasWodka]

Did the substantive pass now the closing-ref is cleared. Both Bugbot findings are real — I verified each against the code, and each has a dimension worth adding. Not duplicating them as my own threads; this is to save you the diagnosis.

---

1. install-cli.sh:290 — the failed-upgrade exit. Confirmed, and the fix is less trivial than it looks.

install_tracebloc_cli ends return 0 unconditionally, so upgrade_cli_onlycannot distinguish success from failure by return code — the if declare -F …; then install_tracebloc_cli; fi; exit 0 shape has no signal to act on even if you wanted one.

The sharper part is your own comment:

# It EXITS the installer (0). Unlike _assess_handoff it does NOT mark the run 'skipped': a newer CLI was installed, so this is a real, succeeded install and telemetry ... records it as such on the 0 exit.

"a newer CLI was installed" is the premise, and on the failure path it is false — so a failed upgrade is recorded as a succeeded install. That's a telemetry lie, not just a cosmetic one, and it will read as "upgrade works fine" in aggregate exactly when it doesn't.

And there's a second half Bugbot didn't name. The failure message inside install_tracebloc_cli is:

warn "Couldn't install the tracebloc CLI automatically — your client is set up fine."

That is correct for its original caller — on the provision path the client genuinely is set up and only the CLI is missing. On tracebloc upgrade it is a non-sequitur: the CLI is the entire thing the user asked for, so reassuring them the client is fine answers a question nobody asked, while the nag they ran the command to clear is still there. A message that was true for one caller becoming false for a new one is the same shape as the stale-remedy fix you made on client#852.

---

2. assess.sh:496 — the early return skipping the drift warnings. Confirmed, and it re-opens a closed bug.

The new return sits immediately above the case, and the very first thing in the healthy arm carries this comment:

# This fast-path hands off and exits before _handle_existing_cluster, so its reuse-path drift check never runs; surface the warning here too so a healthy-but-drifted client still sees the recreate guidance (Bugbot #565).

So that warning exists because an earlier early-exit path skipped it. The new cli-behind-latest return is another early-exit path, above it, and skips it again — reintroducing #565 for the upgrade case. Same for the GPU-consistency guard you added in client#852: a machine whose cluster requests nvidia.com/gpu on a CPU-only node image would run tracebloc upgrade, be told its environment is healthy, and never hear that GPU jobs will sit Pending.

Worth noting this is the third instance of one shape I've hit in this repo today — a correct fix that lands above a guard added to close a previous early-exit gap. client#852's CPU-fallback remedy was the same, and so was frontend-app#909's anchor. It may be worth a check that enumerates the early-exit paths out of assess and asserts each reaches the drift/GPU warnings, rather than fixing them one at a time.

---

On the parts I'm happy with: the cross-repo contract with cli#590 holds on names and semantics (verified separately, note is above), gating _assess_cli_behind_latest on TB_UPGRADE_CLI keeps ordinary runs free of a latest-lookup, and keeping the CLI mutation in main() rather than behind assess.sh's non-mutating contract is the right seam.

Not approving: quality / action-pins is still pending and both Bugbot threads are open — either alone holds it.

[aptracebloc]

bugbot run

[LukasWodka]

0f3ce89e addresses both Bugbot findings and the extra dimension I raised on each. Verified against the diff rather than the commit message.

The drift fix is better than what I'd have suggested. I was thinking about restructuring the early return in assess.sh; calling the advisories from the exit path instead keeps that file a read-only classifier, which was your stated seam:

declare -F _check_existing_cluster_k8s_version >/dev/null 2>&1&& _check_existing_cluster_k8s_version
declare -F _check_healthy_cluster_gpu_consistent >/dev/null 2>&1&& _check_healthy_cluster_gpu_consistent

That also correctly picks up _check_healthy_cluster_gpu_consistent — the guard from client#852 — so an upgrade on a cluster requesting nvidia.com/gpu against a CPU-only node image now gets the same signal the healthy hand-off gives. Guarded with declare -F like the rest of the file, so a stale bootstrap can't reach an undefined function.

The honest-failure fix gets the direction right, and the direction is the interesting part. It exits non-zero only when it can prove failure — both versions parseable and still behind:

if [[ "$latest"=~ ^…$ ]] && [[ "$now"=~ ^…$ ]] && _version_lt "$now""$latest";then
warn "Couldn't update the tracebloc CLI to ${latest} — still on ${now}. …"exit 1
fi

Note this fails the opposite way from _assess_cli_behind_latest, and both are correct: in the classifier, "can't prove you're current" → update, because the user asked for it; here, "can't prove it failed" → exit 0, because crying wolf on an upgrade that may well have worked is its own harm. Two different uncertainty rules in one feature, each pointed the right way, is easy to get wrong and this doesn't.

And the new message fixes the half of the finding Bugbot didn't name. The old text — "your client is set up fine" — was a non-sequitur for this caller. "Couldn't update the tracebloc CLI to 0.10.8 — still on 0.10.5. The update reminder will keep showing until it succeeds." tells the user what failed, what they still have, and what they'll keep seeing. That's the whole answer.

The tests cover the decision's full domain, which is what makes the new conditional trustworthy rather than plausible: still-behind → non-zero; TB_CLI_LATEST unset → 0; unreadable version → 0. That's every branch of the guard, derived from its own structure.

Not approving yet — 41 checks are still running on the new head, and both Bugbot threads are still open. Per that gate's own instructions the threads need resolving and the check re-running (a push isn't required, and an empty commit won't do it). Nothing outstanding from me.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0f3ce89. Configure here.}

[LukasWodka]

Approving 0f3ce89e — 44 passing, 4 path-skipped, nothing pending or failing, both Bugbot threads resolved, mergeable=MERGEABLE. Head is the commit I verified last pass, so nothing needed re-checking.

Recording what settled it, since this went through two rounds:

Both Bugbot findings were real — I checked each against the code rather than relaying them, and each had a dimension worth adding. The exit-0-on-failure one mattered because install_tracebloc_cli returns 0 unconditionally, so there was no signal to act on, and your own comment carried the false premise ("a newer CLI was installed, so this is a real, succeeded install") that would have made a failed upgrade a succeeded install in telemetry. The early-return one re-opened Bugbot #565 for the upgrade path and skipped the GPU guard from client#852 with it.

Your fixes were better than what I'd have proposed. Calling the advisories from the exit path keeps assess.sh a read-only classifier — the seam you'd already chosen — where I'd been thinking about restructuring the return. And the honest-failure check exits non-zero only on a provable failure, which is deliberately the opposite uncertainty rule from _assess_cli_behind_latest in the same feature. Both point the right way: the classifier resolves doubt toward updating because the user asked for it; this resolves doubt toward silence because crying wolf on an upgrade that probably worked is its own harm. Two inverse fail-safes in one change is easy to get backwards.

And the new message fixes the half Bugbot didn't name."Couldn't update the tracebloc CLI to 0.10.8 — still on 0.10.5. The update reminder will keep showing until it succeeds" replaces a line that told the user their client was set up fine — true for the original caller, a non-sequitur here where the CLI is what was asked for.

The tests cover every branch of the new guard: still-behind → non-zero, TB_CLI_LATEST unset → 0, unreadable version → 0.

Merge order, since this is half a cross-repo pair with cli#590 (which I approved): either order is safe. If #590 lands first the CLI passes a flag this installer doesn't yet read — an ignored env var, so today's no-op rather than a regression. If this lands first the installer honours a flag nothing sends yet, which is inert. The fix takes effect when the second one lands.

Value:tracebloc upgrade on a healthy-but-behind machine finally does the one thing its name promises — and when the download fails, says so instead of reporting success and leaving the nag in place.