Skip to content

doctor: reachability gate — stop the cluster-down cascade (misleading remedies) #350

Description

@LukasWodka

Part of epic tracebloc/backend#1142 (Theme A).

HIGH.tb doctor runs all 8 downstream cluster checks even when the API is unreachable (internal/doctor/doctor.go:110-131 — "the parent release gates nothing"). Root mechanism: getDeployment/findDeployment/jobsManagerEnv (doctor.go:592) collapse a transport error and genuine absence into the same nil/empty, so each downstream check invents a wrong verdict/remedy:

  • checkPVC:281 → ✖ "usable StorageClass / PVC Bound" (storage red herring)
  • checkRequestsProxy:371 → ✖ "requests-proxy deployment not found → reinstall chart"
  • checkProxy:303 → ⚠ "chart too old / configMap ref"
  • checkNodeFit:398, checkImagePull:476, checkPods/checkRestartHistory:164 → RBAC/"couldn't read" warnings
  • checkReachable:147 → single generic remedy; no endpoint named

Fix: classify relErr in Run() (transport/connectivity vs cluster.ErrNoParentRelease vs 403/NotFound). When unreachable: emit checkReachable (the one honest ✖) and render the 7 cluster-API checks with a new StatusUnknown ("could not check — cluster API unreachable") and no per-domain remedy; keep checkBackendEgress (doesn't touch the cluster API). checkReachable names the endpoint ("the cluster API at isn't answering — is the cluster running?"; loopback → "start Docker Desktop, then k3d cluster start"). Worst() ignores StatusUnknown so exit code stays from checkReachable. Preserve today's per-check messaging when the API IS reachable (isolated failures).

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions