fix(#431): idempotent GPU reconfigure — no cluster bounce on re-run by shujaatTracebloc · Pull Request #508 · tracebloc/client

shujaatTracebloc · 2026-07-31T06:55:28Z

#431 — idempotent GPU reconfigure (no Docker restart / cluster bounce on re-run)

On a Tier-1/2 GPU host, every re-run of install_nvidia_container_toolkit unconditionally ran nvidia-ctk runtime configure --runtime=docker, then systemctl restart docker, sleep 3, and a CUDA smoke test (docker run --gpus all nvidia/cuda:…). Restarting Docker takes the live k3d cluster down mid-reinstall — a correctness hazard, not just wasted time — and the smoke test re-pulls the CUDA image whenever it was evicted.

Fix (`scripts/lib/gpu-nvidia.sh`)

Skip-when-satisfied. Reconfigure + restart Docker only when the running daemon isn't already defaulting to the NVIDIA runtime (_docker_default_runtime_is_nvidia, via docker info). A re-run on a configured host does nothing here — no restart, no cluster bounce, no image pull.
Deliberate restart when a change IS needed. If the config actually changes and the cluster is live (_k3d_cluster_running, jq-free), it warns that the cluster will restart, restarts Docker, then k3d cluster start brings the cluster back deterministically rather than relying only on the nodes' restart policy.
Cached smoke test. Records the toolkit+driver signature (_gpu_stack_signature) of the last passing test; skips the CUDA pull when unchanged, re-verifies when the toolkit or driver changes.

Bonus (found by the tests)

Fixed a latent awk bug: an exit 0 in a main rule still runs END, so the old END { exit 1 } overrode a positive result. The running-cluster check now decides only in END.

Acceptance

Re-run on an already-configured GPU host: no Docker restart, no cluster bounce, no image pull.
A genuinely changed config still applies cleanly (and the live cluster is brought back).

Tests

New scripts/tests/gpu-nvidia.bats (13 tests): the three helpers + the acceptance behaviours — already-nvidia → no reconfigure/restart; not-nvidia → reconfigure + restart; live cluster → warn + k3d cluster start; cached smoke test → no pull; first run → records the signature. bash -n + shellcheck --severity=error clean; check-style + check-drift clean; scripts/manifest.sha256 regenerated (R8). CI discovers the new bats file via its scripts/tests/*.bats glob.

Closes#431

Note

Medium Risk
Touches installer paths that restart Docker and bring k3d back—high impact when mis-probed—but changes are guarded with bounded probes, explicit recovery, and broad bats coverage.

Overview
Makes install_nvidia_container_toolkit safe to re-run on hosts that already use the NVIDIA Docker runtime, so a repeat install no longer restarts Docker (which drops a live k3d cluster) or re-pulls the CUDA smoke-test image unless something actually changed.

Skip-when-satisfied: New helpers probe the running daemon (docker info via bounded _bounded) and only run nvidia-ctk runtime configure + systemctl restart docker when the default runtime is not already nvidia.

When a restart is required: A tri-state _k3d_cluster_running check (running / not running / unknown on probe failure) drives warnings and a bounded k3d cluster start … --wait --timeout recovery path; cluster start failures are surfaced with a manual hint instead of being swallowed.

Smoke-test cache: A toolkit+driver signature is stored in ~/.tracebloc/.gpu-smoke-ok (under HOST_DATA_DIR); matching signatures skip docker run unless this run reconfigured Docker, in which case the GPU path is re-verified and failed runs clear the marker.

Adds scripts/tests/gpu-nvidia.bats for helpers and acceptance cases; updates scripts/manifest.sha256 for gpu-nvidia.sh.

^{Reviewed by Cursor Bugbot for commit 534403c. Bugbot is set up for automated code reviews on this repo. Configure here.}

…ounce on re-run On a GPU host, every re-run of install_nvidia_container_toolkit unconditionally ran `nvidia-ctk runtime configure --runtime=docker`, then `systemctl restart docker`, then a CUDA smoke test that re-pulls nvidia/cuda. Restarting Docker takes the live k3d cluster DOWN mid-reinstall (a correctness hazard, not just wasted time). - Skip-when-satisfied: reconfigure + restart Docker only when the running daemon isn't ALREADY defaulting to the NVIDIA runtime (_docker_default_runtime_is_nvidia via `docker info`). A re-run on a configured host does nothing here — no restart, no cluster bounce. - When a restart IS needed and the cluster is live, say so and sequence it: warn, restart Docker, then `k3d cluster start` to bring the cluster back deterministically rather than relying only on the nodes' restart policy. Cluster-running is detected jq-free via _k3d_cluster_running. - Cache the smoke test: record the toolkit+driver signature (_gpu_stack_signature) of the last PASSING test; skip (no image pull) when unchanged, re-verify when it changes. Also fixes a latent awk bug the tests caught: an `exit 0` in a main rule still runs END, so the old `END { exit 1 }` overrode a positive result — the running-cluster check now decides only in END. Tests: new scripts/tests/gpu-nvidia.bats — the three helpers plus the acceptance behaviors (already-nvidia -> no reconfigure/restart; not-nvidia -> reconfigure + restart; live cluster -> warn + k3d cluster start; cached smoke test -> no pull; first run -> records the signature). Closes#431

LukasWodka · 2026-07-31T06:55:48Z

👋 Heads-up — Code review queue is at 42 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

.github#112 — promote: cross-repo change guidance (develop -> main) · author: @LukasWodka · no reviewer assigned
averaging-service#255 — chore(ci): enable Dependabot security updates · author: @LukasWodka · no reviewer assigned
backend#1146 — docs(rfc): platform cost & autoscaling architecture · author: @saadqbal · reviewer: @LukasWodka, @saqlainsyed007, @aptracebloc
backend#1201 — docs(rfc): Data spaces & federated dataset composition (draft) · author: @LukasWodka · reviewer: @saadqbal
backend#1217 — docs(rfc): Dataset integrity & score reproducibility (draft) · author: @LukasWodka · reviewer: @saadqbal
backend#1261 — RFC-BACKEND-0008: board & process reset (decision RFC — for markup) · author: @LukasWodka · reviewer: @saadqbal
backend#1284 — docs(deploy): backend deploy runbook skeleton (#1263) · author: @LukasWodka · no reviewer assigned
backend#1332 — feat(check-model): allow KNN/GaussianProcess on a single-edge owned dataset (#923 follow-up) · author: @divyasinghds · no reviewer assigned
backend#1353 — chore(ci): baseline historical gitleaks findings + wire caller · author: @LukasWodka · no reviewer assigned
backend#1361 — feat(tasks): bound ExperimentQueue growth for PAUSED experiments (backend#1351) · author: @saqlainsyed007 · no reviewer assigned

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

LukasWodka · 2026-07-31T07:19:20Z

👋 Heads-up — Code review queue is at 50 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

averaging-service#255 — chore(ci): enable Dependabot security updates · author: @LukasWodka · no reviewer assigned
averaging-service#256 — chore: adopt the org PR template · author: @LukasWodka · no reviewer assigned
backend#1146 — docs(rfc): platform cost & autoscaling architecture · author: @saadqbal · reviewer: @LukasWodka, @saqlainsyed007, @aptracebloc
backend#1201 — docs(rfc): Data spaces & federated dataset composition (draft) · author: @LukasWodka · reviewer: @saadqbal
backend#1217 — docs(rfc): Dataset integrity & score reproducibility (draft) · author: @LukasWodka · reviewer: @saadqbal
backend#1261 — RFC-BACKEND-0008: board & process reset (decision RFC — for markup) · author: @LukasWodka · reviewer: @saadqbal
backend#1284 — docs(deploy): backend deploy runbook skeleton (#1263) · author: @LukasWodka · no reviewer assigned
backend#1332 — feat(check-model): allow KNN/GaussianProcess on a single-edge owned dataset (#923 follow-up) · author: @divyasinghds · no reviewer assigned
backend#1353 — chore(ci): baseline historical gitleaks findings + wire caller · author: @LukasWodka · no reviewer assigned
backend#1361 — feat(tasks): bound ExperimentQueue growth for PAUSED experiments (backend#1351) · author: @saqlainsyed007 · reviewer: @aptracebloc

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

… reconfigure Bugbot round on the idempotent GPU work: - High: _docker_default_runtime_is_nvidia ran a bare `docker info` with no timeout — a wedged daemon would hang a headless re-run at the skip gate. Route it (and the k3d probe) through _bounded (timeout(1)/gtimeout(1)) with TB_PROBE_TIMEOUT. - Medium: after warning the cluster would restart, `k3d cluster start` discarded output and `|| true`'d failure, so a failed bring-up was reported as success. Capture output, and on failure warn with the k3d error + a manual-start hint. - Medium: the smoke-test cache keyed only on toolkit+driver signature, so a re-run that ACTUALLY reconfigured + restarted Docker could still skip `docker run --gpus all` and report a possibly-broken post-restart path as verified. Force the smoke test (bypass the cache) whenever the runtime was reconfigured this run. Tests: bounded-probe source guard; a failed cluster restart is surfaced (warn + manual hint); a reconfigure re-verifies even with a matching marker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The `k3d cluster start` added to bring the cluster back after a Docker restart had no deadline; k3d defaults to --wait with timeout 0 (forever), so a slow post-restart bring-up could hang a headless re-run. Add --wait --timeout "${min}m" (TB_CLUSTER_START_TIMEOUT_MIN, default 5) so it aborts with a real k3d error — which the existing failure path then surfaces (warn + manual-start hint). Mirrors the bounded `cluster create` (#426). Source guard added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…r stale smoke marker Bugbot round 3: - Medium: a forced smoke re-verify that FAILS left the old .gpu-smoke-ok marker in place, so the next (reconfigured=0) run saw a matching signature and skipped the test, treating a failing stack as verified. Remove the marker on smoke failure. - Medium: _k3d_cluster_running returned "not running" both when the cluster was genuinely down AND when the bounded probe failed/timed out. On a wedged daemon (which also fails the nvidia check) that meant Docker was restarted WITHOUT a cluster start, silently leaving a live cluster down. Make it tri-state (0 running / 1 not running / 2 unknown); the caller now attempts recovery on running OR unknown, warning "Couldn't confirm … will try to bring it back". Tests: probe failure -> unknown (rc 2); unknown state still attempts recovery + warns; a failed forced smoke test clears the stale pass marker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…et -euo) Bugbot round 4 — the idempotency helpers tripped the installer's `set -e`: - High: `_k3d_cluster_running; cr=$?` (and the internal `out=$(...); rc=$?`) are bare sequences, so a non-zero probe aborts the install before `systemctl restart docker` on a first-run GPU host (no cluster yet, probe returns 1) — and made the UNKNOWN(2) timeout path unreachable. Use `|| cr=$?` / `|| rc=$?` so the code is captured, not fatal. - Medium: `_gpu_stack_signature` ended on a failed `[[ -n … ]] && printf`, returning 1 when no versions are found; `gpu_sig="$(…)"` then aborted under set -e. Add `return 0` so an empty signature means "don't cache", not "abort". Tests: two set -e integration tests (source common.sh + gpu-nvidia.sh under `set -euo pipefail`) — empty signature and a failed cluster probe both complete without aborting. These exercise the exact context the unit tests (no set -e) missed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

_gpu_stack_signature shelled out to nvidia-ctk + nvidia-smi with no deadline, and it runs at the smoke-test skip gate on every re-run — a half-ready driver (installed without reboot) or a stuck device node could hang a headless re-run, the same class already bounded for docker info / k3d cluster list. Route both through _bounded "${TB_PROBE_TIMEOUT:-5}"; an empty signature on timeout already means "don't cache". Source guard extended to assert both signature probes are bounded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1917877. Configure here.}

The two _gpu_stack_signature tests didn't mock `has`, so on a runner that HAS timeout(1) (CI) the newly-_bounded probes exec the real (absent) nvidia-ctk/nvidia-smi via `timeout` — which can't see shell-function mocks — and the "combines versions" assertion failed (it passed on macOS, which lacks timeout, so _bounded took its passthrough branch). Add `has() { return 1; }` to both tests so _bounded runs the function mocks, matching how the docker/k3d probe tests already suppress timeout. Verified with a fake `timeout` on PATH: 21/21 pass. Production is unaffected (real GPU hosts have the binaries); this was purely a test-mocking gap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

LukasWodka · 2026-08-02T09:03:03Z

/fr-pass

shujaatTracebloc self-assigned this Jul 31, 2026

shujaatTracebloc marked this pull request as ready for review July 31, 2026 07:19

shujaatTracebloc requested a review from saadqbal as a code owner July 31, 2026 07:19