feat(#90): cluster doctor — node-fit + image-pull checks (WS3 follow-up)

[saadqbal]

What

Adds two read-only checks to tracebloc cluster doctor (follow-up to #89). Closes part of #90.

Check	What it catches
Node capacity	A training job can't fit any node → it sits Pending forever
Image pull secret	A missing/malformed registry secret → private images ImagePullBackOff

How

• Node capacity — parses the resource requests the jobs-manager stamps on spawned training jobs (RESOURCE_REQUESTS like cpu=2,memory=8Gi, GPU_REQUESTS like nvidia.com/gpu=1) from its env, and checks at least one Ready node's Allocatable can fit them. GPU is soft: a hard ✖ only when no node satisfies cpu/mem; a ⚠ when a GPU is requested but no node exposes it (jobs-manager has a GPU→CPU fallback, so that's not fatal). Missing/unreadable RESOURCE_REQUESTS or nodes → ⚠ (skip), never a false ✖.
• Image pull secret — reads the jobs-manager's imagePullSecrets; if it references one, verifies the secret exists, is kubernetes.io/dockerconfigjson, and its .dockerconfigjson is non-empty valid JSON. No pull secret in use → ✔ (public/digest-pinned). (Bad-but-well-formed credentials can't be detected without an actual pull — noted; this catches the common missing/empty/malformed misconfig.)

Both fit doctor's existing read-only, best-effort, injectable-clientset design.

Out of scope (tracked on #90)

The in-cluster egress probe — the third deferred check — is intentionally a separate PR. Today's Backend egress check probes from the CLI host, not the cluster's egress path; a real probe needs a port-forward to the egress-proxy's probe endpoint (or an exec/probe pod), which is a different mechanism from these read-only checks and deserves its own design.

Testing

make ci green (vet, go test -race -cover, errcheck, ineffassign, misspell, gofmt -s, schema-check). New table tests with the fake clientset cover node-fit (fits / no-fit / GPU-missing-warn / not-ready-node / missing-requests-warn), image-pull (none / valid / missing / malformed), and the parseCPUMem/parseGPU helpers. TestRun_HealthyCluster updated to 8 checks.

🤖 Generated with Claude Code

---

Note

Low Risk
Read-only diagnostic checks using existing Kubernetes list/get APIs; no cluster mutations or auth changes.

Overview
Extends tracebloc cluster doctor with two read-only checks (8 total), aimed at jobs that stay Pending or hit ImagePullBackOff.

Node capacity reads jobs-manager RESOURCE_REQUESTS / GPU_REQUESTS, lists Ready nodes, and requires one node whose allocatable covers cpu+memory (and GPU when requested). GPU-only gaps are warn (CPU fallback); missing env or node list permission is warn, not fail. Logic explicitly avoids treating cpu/mem on one node plus GPU on another as schedulable (Bugbot #91).

Image pull secret inspects jobs-manager imagePullSecrets: absent refs → ok; otherwise each secret must exist, be kubernetes.io/dockerconfigjson, and have valid non-empty .dockerconfigjson JSON.

Adds parsing helpers (parseResourceSpec, parseCPUMem, parseGPU, nodeReady) and fake-clientset tests for both checks; TestRun_HealthyCluster now supplies RESOURCE_REQUESTS and a fitting node.

^{Reviewed by Cursor Bugbot for commit c04ab62. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a0bb000. Configure here.}

[aptracebloc]

Non-blocking review notes — one actionable nit + one scope clarification. (Nice that it already carries the httpProbeClose() fix from #89.)

Map-iteration nondeterminism — parseGPU:

fork, v:=rangeparseResourceSpec(spec) {
ifq, err:=resource.ParseQuantity(v); err==nil&&!q.IsZero() {
returncorev1.ResourceName(k), q, true
}
}

Go randomizes map iteration order, so if GPU_REQUESTS ever carried more than one entry this returns a nondeterministic one (and risks a flaky test). Single-entry specs (nvidia.com/gpu=1) are fine today; cheap hardening is to iterate sorted keys, or read the expected key directly.

Scope clarification (not a bug) — capacity vs. free capacity:
checkNodeFit compares the request against each node's Allocatable (total schedulable capacity), not allocatable minus already-requested pods. So it answers "is any node fundamentally big enough?" — the right question for a doctor — but it won't catch "nodes are big enough but currently full." Worth a word in the detail/remedy so a ✔ isn't read as "schedulable right now."

The per-node fit logic (requiring cpu+mem+GPU on a single node rather than OR-ing across nodes) is the subtle-but-correct call here — good that it's pinned by tests.

— drafted with Claude (Opus 4.8), sent by @aptracebloc