Uh oh!
There was an error while loading. Please reload this page.
fix(tests): isolate assess healthy-classify test from the host tracebloc CLI (backend#2700) - #882
Conversation
…loc CLI (backend#2700) The `_assess_classify: all signals true -> healthy` test stubbed every leaf probe except `_assess_cli_outdated`, so the classify path shelled out to the real `tracebloc version` on the machine running bats. On Linux CI (no CLI on PATH) that fails open to "not outdated" and the test reaches healthy; on a machine with a real below-floor CLI (e.g. 0.7.0 in ~/.local/bin) the floor check correctly fires cli-outdated, so the box classifies degraded and the healthy assertion fails. Not an assess.sh bug — a below-floor CLI IS cli-outdated. Stub the leaf to "above the floor" (return 1), matching the sibling classify tests (assess.bats:309/323/341/361/380), so the test no longer depends on whatever tracebloc happens to be installed on the host. Test-only; no scripts/lib change (no manifest regen / CODEOWNERS review). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
saqlainsyed007
left a comment
There was a problem hiding this comment.
Approving at 55b36e6. Correct test-isolation fix, no findings.
The "all signals true -> healthy" classify test stubbed every leaf except _assess_cli_outdated, so it shelled out to the host's real tracebloc version: green on CI (no CLI on PATH, fails open to not-outdated) but flaky on a machine with a below-floor CLI in ~/.local/bin (which correctly classifies cli-outdated and masks healthy). Stubbing the leaf to return 1 (above the floor) — the same stub the sibling classify tests at assess.bats:309/323/341/361/380 use — makes the aggregation test deterministic without weakening it: cli-outdated detection is exercised by its own tests, and this test is asserting the all-healthy → healthy aggregation, so isolating the leaf is right. Test-only, no scripts/lib change (no manifest regen). CI green, mergeable, no threads. LGTM.
Uh oh!
There was an error while loading. Please reload this page.
Closes tracebloc/backend#2700
What
scripts/tests/assess.bats→_assess_classify: all signals true (all three workloads Ready + CLI) -> healthy(assess.bats:583) was a flaky, environment-dependent test. Green on Linux CI, but fails on any machine that has a realtraceblocCLI below the0.10.0floor on PATH / in~/.local/bin(reproduced on macOS bash 3.2.57 with a realtracebloc 0.7.0). Pre-existing; unrelated to backend#2685 / #874.Root cause — test-isolation leak, not an
assess.shbugThe test stubs every leaf probe (
has,_cluster_exists,_assess_cluster_servers_running,_assess_release_pending,kubectl) except_assess_cli_outdated, so_assess_classifyreaches the floor check and shells out to the realtracebloc version:traceblocon PATH →tracebloc versionfails →_assess_cli_outdatedfails open ("not outdated") → classify reacheshealthy. ✅0.7.0 < 0.10.0→_assess_cli_outdatedcorrectly returns "outdated" → classify returnsdegraded/cli-outdated, and the[ "$INSTALL_STATE" = healthy ]assertion fails. ❌assess.shis correct in both cases — a below-floor CLI iscli-outdated. Every sibling classify test that reaches this branch already stubs_assess_cli_outdated(assess.bats:309/323/341/361/380); the healthy test was the only one that didn't. Not a_bounded/timeout(1)issue —_assess_runtime_downis stubbed insetup()and the classify path never touches coreutilstimeout.Fix
Add
_assess_cli_outdated() { return 1; }(above the floor) to the healthy test, matching the direct sibling at assess.bats:315. Test-only — noscripts/lib/change, so nomanifest.sha256regen and no CODEOWNERS review (/scripts/tests/isn't owned).Verification (macOS, bash 3.2.57 arm64, no timeout/gtimeout on PATH)
bats scripts/tests/assess.batsbats scripts/tests/cluster.batsscripts/check-style.shscripts/gen-manifest.sh --check🤖 Generated with Claude Code
Note
Low Risk
Test isolation only in scripts/tests; no production assess.sh or lib changes.
Overview
Fixes environment-dependent failure in the
_assess_classify“all signals true → healthy” bats case by stubbing_assess_cli_outdatedso classification does not invoke the hosttracebloc version.Without that stub, a real CLI below the
0.10.0floor correctly yieldsdegraded/cli-outdated, which breaks the test’shealthyassertion on dev machines (e.g. macOS with0.7.0in~/.local/bin) while Linux CI often passes because no CLI is on PATH. Sibling classify tests already stub this leaf; this change aligns the healthy-path test with them.Test-only change in
assess.bats— no installer logic changes.Reviewed by Cursor Bugbot for commit 55b36e6. Bugbot is set up for automated code reviews on this repo. Configure here.