feat(#88): tracebloc cluster doctor — live-cluster health checks (WS3)

[saadqbal]

What

Adds tracebloc cluster doctor — a read-only health sweep of a running tracebloc client cluster that prints a ✔/⚠/✖ line per check with a remedy for anything that isn't green.

Closes#88. Parent epic: client-runtime#116 (WS3 — diagnose without tracebloc access).

Why

When a customer's experiment silently sits in Pending, today there's no self-serve way to ask "is my cluster healthy?" — the cluster info command answers "am I pointed at the right cluster?" but not "is it working?". doctor fills that gap, over the customer's normal kubeconfig.

It's the sibling cluster info's own doc comment already anticipated:

Future verbs (e.g. cluster doctor for diagnostics) hang off this parent in later phases.

The 6 checks (lean MVP)

Check	What it verifies
Cluster reachable	API answers + parent client release discovered
Pod health	nothing crash-looping / stuck Pending (local complement to client-runtime#117)
Dataset volume	shared PVC exists + Bound (reuses cluster.DiscoverSharedPVC)
Proxy configuration	in-cluster requests/egress proxy wiring (the corporate-proxy propagation class)
Backend egress	tracebloc backend reachable (host-side, proxy-aware)
Service Bus egress	requests-proxy Ready — the broker for the experiments queue; while it's down, experiments stay Pending

Ends with a consolidated verdict + a pointer to ./install-k8s.sh --diagnose for a full support bundle.

How

• internal/cli/doctor.go — thin cobra command, sibling of cluster info, same kubeconfig/context/namespace flags; renders results via ui.Printer and maps the worst status to an exit code.
• internal/doctor/ — standalone package: Status/Result/Worst + the 6 checks. Reuses cluster.Load/NewClientset/DiscoverParentRelease/DiscoverSharedPVC. Network probes are injectable (Options.HTTPProbe) so it's fully testable without a real cluster or egress.
• Every check is independent and best-effort — one failure never hides the others (mirrors the installer's preflight.sh contract).

Honest scope notes

• Backend egress is probed from the machine the CLI runs on, not from inside the cluster (the cluster egresses via its egress-proxy). It still catches a customer network/proxy that can't reach the backend at all; a true in-cluster probe is the WS3 follow-up.
• support-bundle already ships as the installer's install-k8s.sh --diagnose — not rebuilt here.
• Deferred to a follow-up: node-resources-vs-spawned-job-request fit, and image pullability.

Exit codes

0 all passed (or warnings only) · 2 one or more checks failed · 3 kubeconfig couldn't be loaded.

Testing

make ci green: vet, go test -race -cover, errcheck, ineffassign, misspell, gofmt -s, schema-check. New internal/doctor package is 82.2% covered (table-driven, client-go fake clientset + injected probes). Manually verified tracebloc cluster doctor --help and command registration under cluster.

🤖 Generated with Claude Code

---

Note

Low Risk
Read-only Kubernetes inspection and optional outbound HTTP probes; no changes to auth, ingestion, or cluster mutation paths.

Overview
Adds tracebloc cluster doctor under the existing cluster command — a read-only post-install sweep that prints ✔/⚠/✖ per check with remedies, complementing cluster info (targeting vs health).

The CLI loads kubeconfig the same way as cluster info, runs internal/doctor, and maps the worst check to exit codes 0 (ok/warn), 2 (fail), or 3 (kubeconfig).

Six independent checks always run (failures don’t hide others): parent release reachable, pod crash-loop / long-Pending, shared PVC Bound, jobs-manager proxy env, backend HTTPS reachability from the user’s machine (injectable probe), and requests-proxy ready — with findDeployment tied to the discovered Helm release to avoid false greens across multiple installs.

Table-driven tests use the fake clientset and injected HTTP probes (~82% coverage on the new package).

^{Reviewed by Cursor Bugbot for commit c9fa17b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[aptracebloc]

Post-approval follow-up (non-blocking — already approved 👍): one small doc fix + two optional nits.

const vs var mismatch — internal/doctor/doctor.go:
The tunables comment says they're "exported-as-vars (not consts) so a future flag could override them," but they're declared as unexported const:

const (
pendingGrace=5*time.MinutehttpProbeTimeout=8*time.Second
)

A const can't be overridden at runtime (not addressable / can't be reassigned), so the rationale doesn't hold as written. Either reword the comment to "conservative package consts," or — if override is actually intended — make them Options fields (e.g. PendingGrace time.Duration) so it threads through the existing Run(opts) seam rather than package-level vars.

Minor (optional):

• checkProxy reads only literal env from jobs-manager (jobsManagerEnv skips valueFrom), so a chart that sets REQUESTS_PROXY_URL via a configMap/secret ref would false-WARN. Worth a line in the remedy, or accept as a known edge.
• httpProbe returns resp.Body.Close() as the probe result; defer resp.Body.Close(); return nil avoids reporting "unreachable" on a (rare) close error after a successful connect.

Really clean package otherwise — the injectable HTTPProbe + fake-clientset tests make it a solid template for future cluster <verb> siblings.

— drafted with Claude (Opus 4.8), sent by @aptracebloc

[saadqbal]

Thanks @aptracebloc — all three addressed in 289f237 (non-blocking, but cheap + correct):

1. const/var comment — reworded: they're conservative package consts; the comment now points at Options (like HTTPProbe) as the seam for any future runtime tuning, rather than implying these are overridable vars.
2. checkProxyvalueFrom — the WARN detail now calls out that a REQUESTS_PROXY_URL set via a configMap/secret ref reads as empty here (we read only literal env), so a ref-based install isn't mistaken for missing wiring.
3. httpProbe close error — a successful client.Do means the host is reachable; we now _ = resp.Body.Close(); return nil so a rare post-connect close error isn't reported as "unreachable" (kept the explicit discard to stay errcheck-clean rather than defer).

make ci still green. Appreciate the careful read 🙏

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 86787e2. Configure here.}