Skip to content

fix(tests): isolate assess healthy-classify test from the host tracebloc CLI (backend#2700) - #882

Merged
aptracebloc merged 1 commit into
developfrom
fix/2700-assess-classify-cli-outdated-isolation
Aug 27, 2026
Merged

fix(tests): isolate assess healthy-classify test from the host tracebloc CLI (backend#2700)#882
aptracebloc merged 1 commit into
developfrom
fix/2700-assess-classify-cli-outdated-isolation

Conversation

@aptracebloc

@aptraceblocaptracebloc commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Closes tracebloc/backend#2700

What

scripts/tests/assess.bats_assess_classify: all signals true (all three workloads Ready + CLI) -> healthy (assess.bats:583) was a flaky, environment-dependent test. Green on Linux CI, but fails on any machine that has a real tracebloc CLI below the 0.10.0 floor on PATH / in ~/.local/bin (reproduced on macOS bash 3.2.57 with a real tracebloc 0.7.0). Pre-existing; unrelated to backend#2685 / #874.

Root cause — test-isolation leak, not an assess.sh bug

The test stubs every leaf probe (has, _cluster_exists, _assess_cluster_servers_running, _assess_release_pending, kubectl) except _assess_cli_outdated, so _assess_classify reaches the floor check and shells out to the real tracebloc version:

  • Linux CI — no tracebloc on PATH → tracebloc version fails → _assess_cli_outdated fails open ("not outdated") → classify reaches healthy. ✅
  • A host with a below-floor CLI0.7.0 < 0.10.0_assess_cli_outdated correctly returns "outdated" → classify returns degraded/cli-outdated, and the [ "$INSTALL_STATE" = healthy ] assertion fails. ❌

assess.sh is correct in both cases — a below-floor CLI iscli-outdated. Every sibling classify test that reaches this branch already stubs _assess_cli_outdated (assess.bats:309/323/341/361/380); the healthy test was the only one that didn't. Not a _bounded/timeout(1) issue — _assess_runtime_down is stubbed in setup() and the classify path never touches coreutils timeout.

Fix

Add _assess_cli_outdated() { return 1; } (above the floor) to the healthy test, matching the direct sibling at assess.bats:315. Test-only — no scripts/lib/ change, so no manifest.sha256 regen and no CODEOWNERS review (/scripts/tests/ isn't owned).

Verification (macOS, bash 3.2.57 arm64, no timeout/gtimeout on PATH)

CheckResult
bats scripts/tests/assess.bats82/82 pass
bats scripts/tests/cluster.bats170/170 pass
scripts/check-style.shclean
scripts/gen-manifest.sh --checkup to date

🤖 Generated with Claude Code


Note

Low Risk
Test isolation only in scripts/tests; no production assess.sh or lib changes.

Overview
Fixes environment-dependent failure in the _assess_classify “all signals true → healthy” bats case by stubbing _assess_cli_outdated so classification does not invoke the host tracebloc version.

Without that stub, a real CLI below the 0.10.0 floor correctly yields degraded/cli-outdated, which breaks the test’s healthy assertion on dev machines (e.g. macOS with 0.7.0 in ~/.local/bin) while Linux CI often passes because no CLI is on PATH. Sibling classify tests already stub this leaf; this change aligns the healthy-path test with them.

Test-only change in assess.bats — no installer logic changes.

Reviewed by Cursor Bugbot for commit 55b36e6. Bugbot is set up for automated code reviews on this repo. Configure here.

…loc CLI (backend#2700)
The `_assess_classify: all signals true -> healthy` test stubbed every leaf
probe except `_assess_cli_outdated`, so the classify path shelled out to the
real `tracebloc version` on the machine running bats. On Linux CI (no CLI on
PATH) that fails open to "not outdated" and the test reaches healthy; on a
machine with a real below-floor CLI (e.g. 0.7.0 in ~/.local/bin) the floor
check correctly fires cli-outdated, so the box classifies degraded and the
healthy assertion fails.
Not an assess.sh bug — a below-floor CLI IS cli-outdated. Stub the leaf to
"above the floor" (return 1), matching the sibling classify tests
(assess.bats:309/323/341/361/380), so the test no longer depends on whatever
tracebloc happens to be installed on the host. Test-only; no scripts/lib
change (no manifest regen / CODEOWNERS review).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@aptraceblocaptracebloc self-assigned this Aug 27, 2026
@LukasWodka
LukasWodka requested review from saqlainsyed007 and removed request for saadqbalAugust 27, 2026 12:58

@saqlainsyed007saqlainsyed007 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 55b36e6. Correct test-isolation fix, no findings.

The "all signals true -> healthy" classify test stubbed every leaf except _assess_cli_outdated, so it shelled out to the host's real tracebloc version: green on CI (no CLI on PATH, fails open to not-outdated) but flaky on a machine with a below-floor CLI in ~/.local/bin (which correctly classifies cli-outdated and masks healthy). Stubbing the leaf to return 1 (above the floor) — the same stub the sibling classify tests at assess.bats:309/323/341/361/380 use — makes the aggregation test deterministic without weakening it: cli-outdated detection is exercised by its own tests, and this test is asserting the all-healthy → healthy aggregation, so isolating the leaf is right. Test-only, no scripts/lib change (no manifest regen). CI green, mergeable, no threads. LGTM.

@aptracebloc
aptracebloc merged commit da87978 into developAug 27, 2026
42 checks passed
@aptracebloc
aptracebloc deleted the fix/2700-assess-classify-cli-outdated-isolation branch August 27, 2026 14:42
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@aptracebloc@saqlainsyed007@LukasWodka