Skip to content

fix(installer): bound healthy-path GPU-consistency Helm calls on macOS (backend#2685) - #874

Merged
aptracebloc merged 1 commit into
developfrom
fix/2685-bound-healthy-gpu-helm-macos
Aug 27, 2026
Merged

fix(installer): bound healthy-path GPU-consistency Helm calls on macOS (backend#2685)#874
aptracebloc merged 1 commit into
developfrom
fix/2685-bound-healthy-gpu-helm-macos

Conversation

@aptracebloc

@aptraceblocaptracebloc commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

What

_check_healthy_cluster_gpu_consistent (the client#835 healthy fast-path GPU guard, run on assess hand-off and on tracebloc upgrade) wraps helm list and helm get values in _bounded. On a stock Mac_bounded runs the bare command — neither timeout(1) nor gtimeout(1) ships there (both are GNU coreutils) — and Helm has no --request-timeout of its own. So a healthy re-run could hang with no further output against a wedged kube-apiserver, even though the function's own header claimed "Bounded".

Deferred Cursor Bugbot Medium from the refused client#869 → staging hop (thread), filed as backend#2685.

Fix

Gate the Helm calls on a self-boundingkubectl cluster-info --request-timeout=5s reachability probe (client-side, coreutils-free) before shelling out to Helm — the exact gate diagnose.sh already puts in front of its own unbounded Helm calls. An unreachable API just means we can't tell, which is already this guard's silent no-op.

  • The later kubectl get nodes already carries --request-timeout=5s.
  • The docker inspect keeps _bounded (matching detect_gpu's sibling inspect at ~L912) and is reached only after this probe + a successful bounded kubectl get nodes prove the docker-hosted k3d API answers — the daemon is provably live sub-second earlier. Deliberately not diverging to a bespoke bound there; noted in the Bugbot deferral so a re-review sees the reasoning rather than a silent claim of coverage.

Verification

Reproduced and verified on the finding's exact environment — macOS bash 3.2.57, arm64, with neither timeout nor gtimeout on PATH:

  • bats scripts/tests/cluster.bats scripts/tests/gen-manifest.bats190 passing, 0 failing, including a new test: an unreachable API returns 0 without shelling out to Helm.
  • Tightened the AMD / CPU "no node probe" assertions to the node probe specifically (cluster-info now legitimately calls kubectl).
  • scripts/check-style.sh → clean.
  • scripts/gen-manifest.sh --check → up to date (regenerated scripts/manifest.sha256; scripts/lib/ change is @saadqbal CODEOWNERS-gated as expected).

One pre-existing, unrelated macOS failure in assess.bats (_assess_classify: all signals true) is not touched by this change (different file; fails on pristine develop too).

Closes tracebloc/backend#2685


Note

Low Risk
Installer-only change to a diagnostic fast-path with silent no-op on failure; no auth, data, or chart behavior changes.

Overview
On stock macOS, _bounded does not time out Helm (no timeout/gtimeout, and Helm has no --request-timeout), so _check_healthy_cluster_gpu_consistent could hang on helm list / helm get values during a healthy re-run against a wedged API.

The guard now runs kubectl cluster-info --request-timeout=5s first; if the API is unreachable it returns silently and never calls Helm, matching the gate diagnose.sh already uses. Comments document the macOS-safe bound behavior.

cluster.bats adds coverage for unreachable API (no Helm), and narrows assertions so cluster-info is expected while get nodes stays absent for AMD/CPU-only paths.

Reviewed by Cursor Bugbot for commit 5669178. Bugbot is set up for automated code reviews on this repo. Configure here.

…S (backend#2685)
_check_healthy_cluster_gpu_consistent (the client#835 healthy fast-path GPU
guard) wraps `helm list` and `helm get values` in `_bounded`. On a stock Mac
`_bounded` runs the BARE command — neither timeout(1) nor gtimeout(1) ships
there (both are GNU coreutils) — and Helm has no `--request-timeout` of its
own, so a healthy re-run could hang with no output against a wedged
kube-apiserver. The function's own header even claimed "Bounded".
Gate the Helm calls on a self-bounding `kubectl cluster-info
--request-timeout=5s` reachability probe (client-side, coreutils-free) — the
same gate diagnose.sh already puts in front of its unbounded Helm calls. An
unreachable API just means we can't tell, which is already this guard's silent
no-op. The later `kubectl get nodes` already carries --request-timeout=5s; the
`docker inspect` keeps `_bounded` (matching detect_gpu's sibling inspect) and
is reached only after this probe proves the docker-hosted API answers.
Verified on macOS bash 3.2.57 with no timeout/gtimeout on PATH. Adds a test
that an unreachable API returns 0 without shelling out to Helm; tightens the
AMD/CPU no-node-probe assertions to the node probe specifically (cluster-info
now legitimately calls kubectl). Regenerated scripts/manifest.sha256.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@aptraceblocaptracebloc self-assigned this Aug 27, 2026
@aptracebloc
aptracebloc requested review from saqlainsyed007 and removed request for saqlainsyed007August 27, 2026 12:17

@saadqbalsaadqbal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The defect is a function whose header claimed "Bounded" while _bounded degrades to running the bare command on a stock Mac — neither timeout(1) nor gtimeout(1) is there, and Helm has no --request-timeout of its own. A wrapper named for a property it doesn't provide on the platform most of the team develops on is worse than no wrapper, because the name is what stops anyone checking.

Gating on kubectl cluster-info --request-timeout=5s is the right fix for the right reason: it's self-bounding and coreutils-free, so it doesn't depend on the thing that's missing, and it's the same gate diagnose.sh already puts in front of its own unbounded Helm calls rather than a new invention. And "an unreachable API just means we can't tell, which is already this guard's silent no-op" means the failure direction doesn't change — the guard was always allowed to abstain.

Leaving docker inspect on _bounded and writing down why is the honest choice, and it's the part I'd have pushed on otherwise. The argument holds: it's reached only after the reachability probe and a successful bounded kubectl get nodes, so the docker-hosted k3d API answered sub-second earlier and the daemon is provably live. Recording that in the Bugbot deferral so a future re-review sees the reasoning — rather than leaving a silent claim of coverage — is exactly the distinction that matters when the next person greps for remaining _bounded calls.

Verified two things rather than taking them:

manifest.sha256 is genuinely regenerated — it records 4d199369…9c3b for scripts/lib/cluster.sh and the shipped file hashes to exactly that. That check has now mattered three times in this repo, and it fails at install time rather than in CI when it's wrong.

The new test proves the negative behaviourally: helm() is stubbed to emit a should-not-run marker, the API is made unreachable via the cluster-info stub returning 1, and the assertions are status -eq 0 plus nothing surfaced — so a Helm shell-out would show up rather than being assumed absent.

Reproducing on macOS bash 3.2.57, arm64, with neither timeout nor gtimeout on PATH is the only environment where this bug exists, so testing there rather than on Linux is what makes the verification mean anything.

Tightening the AMD/CPU "no node probe" assertions to the node probe specifically — because cluster-info now legitimately calls kubectl — is the collateral precision most PRs skip, and skipping it would have left two tests quietly weaker than their names.

Green, no threads. 👍

@aptracebloc
aptracebloc merged commit 8e47c48 into developAug 27, 2026
51 checks passed
@aptracebloc
aptracebloc deleted the fix/2685-bound-healthy-gpu-helm-macos branch August 27, 2026 12:45
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@aptracebloc@saadqbal@LukasWodka