Uh oh!
There was an error while loading. Please reload this page.
feat(resources): tracebloc resources set — guided compute budget (P2, wizard + GPU) (#143) - #241
Conversation
…tracebloc may use (P1) Adds the top-level `tracebloc resources` command (cli#143): the one-knob, read-only view of how much of this machine tracebloc may use. No Kubernetes vocabulary in the output — one number for the machine, one for tracebloc's per-training-run share. P1 (SHOW) — built: - New pure `internal/resources` package: machine capacity from Ready-node allocatable (summed; single-node installer path is normally one node), per-run ceiling parsed from the jobs-manager RESOURCE_LIMITS env (the same source `cluster doctor`'s checkNodeFit reads, so the two never disagree), chart-default fallback, GPU surfacing, and user-language CPU/GiB formatting. - Release-scoped jobs-manager env reader that mirrors doctor.findDeployment's attribution rule on BOTH branches (release-known: prefixed name or instance-label-matched bare; release-unknown: unique suffix match, else nil) — never reads another release's component. - `internal/cli/resources.go`: bare `tracebloc resources` shows; resolves the cluster via the shared resolveClusterTarget seam (exit 3 kubeconfig / 4 no-release), --verbose adds the raw env + node/GPU breakdown. Wired into root.go's command tree and home screen. - Tests: resources pkg ~97%, internal/cli stays above its coverage floor. P2 (set --cpu/--memory, set max) and P3 (macOS Docker VM raise) — deferred: the shipped groundwork doesn't yet re-expose a safe persistence path to the CLI. `set` must write Helm values (a `kubectl set env` is reverted by the hourly auto-upgrade CronJob), and `helm upgrade` needs the chart reference the installer resolves via TRACEBLOC_HELM_REPO_NAME / a dev path — not recoverable from `helm list`. Rather than shell Helm blindly at a live training cluster, `set` is wired as a proper subcommand — `--cpu`/`--memory` flags and an optional `max` positional in the approved shape — whose RunE returns an honest exit-1 "not supported in this build yet" message that points back at `tracebloc resources`. Wiring the flags now means the designed invocation parses cleanly (no cobra "unknown flag") and P2 slots in behind it. Refs #143 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s machine (P2) Builds the mutating half of `tracebloc resources` (cli#143 P2). The number the user sets is the per-run ceiling written verbatim to the chart's RESOURCE_* env (Decision A); tracebloc's ~1 core / 3 GiB overhead is a fit-check safety margin only, never subtracted. - Interactive wizard by default on a TTY: shows the current per-run budget vs the machine, offers "use as much as possible" (pre-selected), "choose an amount" (bounded per-dimension prompts so over-asking is impossible), or "leave it". - Flags for scripting / non-TTY: --cores (hidden --cpu alias), --memory (GB/G/Gi/GiB/bare → GiB), --gpus, and the `max` positional. Single-dimension sets keep the others. Human validation messages on every bad path; exit 2. - GPU is first-class + whole-unit; omitted entirely on a GPU-less machine (the chart's default GPU env is normalized away so a plain --cores change doesn't fail the GPU fit-check on a CPU-only host). - Fit-check against the largest single Ready node (mirrors doctor.checkNodeFit), overhead included; on macOS the honest ceiling message points at Docker Desktop → Resources (VM auto-raise stays deferred to P3). - Apply via new internal/helm seam mirroring the installer/auto-upgrade idiom: resolve chart ref (or TRACEBLOC_CHART_PATH), version-gate --reset-then-reuse-values, pin --version, temp -f values file (dodges the --set comma footgun), resolved --namespace/--kube-context/--kubeconfig, --wait. --dry-run prints the command + values and runs nothing. Cluster-free tests via the fake clientset, prompter seam, and a fake helm Runner. make ci green. Closes#143 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…ror (Bugbot) Interrupting a wizard prompt surfaced survey's errInteractiveCancelled raw from decideDesired, so bare `tracebloc resources set` + Ctrl-C exited 1 with "Error: cancelled by user" — unlike every other prompting command. Map it to the same clean exit 0 + "Cancelled — nothing was changed." note the confirm decline already prints; validation errors (exit 2) and real terminal failures pass through unchanged. Mutation-proven test: cancelled wizard → nil error, the note, and no helm mutation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
…er fits (Bugbot)
validateDesired ran before the sameCeiling no-op check, so on a machine that
shrank under an already-applied ceiling (smaller Docker Desktop VM, lost
node), "Leave it as it is" — and flags merely restating the current values —
failed with exit 2 instead of the promised no-op ("→ no-op skip downstream").
Check sameCeiling first: leaving things unchanged mutates nothing, so there is
nothing for the fit-check to protect; an actual change is still validated
before anything mutates. sameCeiling is a pure desired-vs-current comparison
and needs nothing validateDesired computes, so the reorder is safe.
Mutation-proven test (flags-restate + wizard leave-as-is succeed with no helm
call on a shrunken machine; a real change still exits 2); make ci green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>Uh oh!
There was an error while loading. Please reload this page.
…(Bugbot) Ctrl-C at the final "Let each training run use up to …?" confirm was routed through mapClientErr — exit 0, but SILENT, so a user could finish the wizard, interrupt the confirm, and walk away believing the change went through. The wizard interrupt, the confirm decline, and data delete all print "Cancelled — nothing was changed."; the confirm interrupt now does the same (exit 0, note, no helm mutation). Real terminal failures still go through mapClientErr. Mutation-proven: reverting to the bare mapClientErr path keeps exit 0 but drops the note, failing TestSet_ConfirmCtrlCIsACleanCancel. make ci green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
aptracebloc
left a comment
There was a problem hiding this comment.
Approving. This mutates a live customer cluster (helm upgrade, jobs-manager rolls), so I looked hardest at the two safety-critical files — and the guardrails are exactly right.
What makes a live-mutation PR safe to approve here
- Unpinned refusal (the key one):
helm.Upgraderefuses to upgrade a remote chart without a pinned--version— checked first, before touching anything, and even in--dry-run(so the printed plan is never an unsafe command). Stops a values-only change from silently bumping the whole chart. - Mutates the resolved cluster (ns/context/kubeconfig from
resolveClusterTarget), never the ambient current-context. --reset-then-reuse-values(version-gated,--reuse-valuesfallback) + temp-fvalues file (dodges thecpu=2,memory=8Gi--setcomma footgun); values%q-quoted; temp file cleaned up.- GPU removal writes explicit empty
GPU_LIMITS/GPU_REQUESTS— not omitted (reset-then-reuse would re-inherit) and not=0(client-runtime reads any non-empty as GPU-present). Grounded in_gpu_available_from_envand round-trips through the package's ownparseGPU. A production-only trap, caught and mutation-proven. - Decision A honesty (
DeriveTrainingis the identity; overhead is a fit-margin only, never secretly subtracted), single-node fit mirroringdoctor.checkNodeFit, and non-technical-user input handling (ParseCoresrejects unit suffixes;ParseMemoryGiBnormalizes to GiB).Overhead()is a function so aQuantity.Addcan't mutate a shared value. - The whole helm path is mutation-proven via the
Runnerseam.
Notes (non-blocking)
- Merge #237 first — this is stacked on
feat/143-resources-command; GitHub retargets todevelopon that merge. - Consider a throwaway-cluster smoke before prod reliance. The tests are excellent but fake-
Runner-based — they prove the argv/values/pin/dry-run, not that real helm accepts the upgrade on a real release. For a live-mutation feature, a small e2e (like #236 did fordelete) would be the reassuring final gate. Worth a follow-up. - Minor confirm:
UpgradeParams.Releaseis documented as "== namespace (installer convention)" — worth confirming it passes the discovered release name; either way a mismatch fails safe (helm: release not found).
Approving as-is — the two production-subtle traps (unpinned bump, GPU re-inherit) were found and mutation-proven, and the live-smoke is a follow-up, not a fix.
— drafted with Claude (Opus 4.8), sent by @aptracebloc
#237 put `resources show` on develop, so un-dash SHOW (keep `set`/#241 dashed), bump the basis commit to develop @27c5392, and reconcile the §4 heading + known-gaps note. The exit-6/--overwrite flow already matches current develop (data.go:236), so it's left as-is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add the CLI navigation map (single source of truth) Four Mermaid flowcharts (top-level, the two gate chains, data ingest, resources) + exit-code legend + cross-links + known gaps. Diffable, renders on GitHub, kept current via PRs. Flags: two independent gate chains; stateless home today (status redesign proposed); delete exits 0 on partial offboard; resources unshipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: nav-map — resources show shipped (#237); rebase on develop #237 put `resources show` on develop, so un-dash SHOW (keep `set`/#241 dashed), bump the basis commit to develop @27c5392, and reconcile the §4 heading + known-gaps note. The exit-6/--overwrite flow already matches current develop (data.go:236), so it's left as-is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Arturo Peroni <arturo@tracebloc.io>
#241 was stacked on #237 (resources SHOW), which squash-merged to develop; develop also gained the #244 status-aware home screen + #231 top-level doctor. Resolved: - internal/cli/resources.go, resources_test.go → #241's (the BUILT `set`). develop's copy is #237's SHOW + the deferred `resources set` stub, and #241 is exactly the PR that replaces that stub with the real command. Only #237 ever touched these files on develop, so no SHOW change is lost. - internal/cli/root.go → develop's (new home screen + top-level doctor); #241 adds nothing to root.go (`resources` is wired via resources.go).
…no-op ceiling (Bugbot #241) Bugbot MED (Arturo flagged for a decision): on a GPU-less machine whose cluster still carries the chart-default GPU_REQUESTS=nvidia.com/gpu=1, restating the current CPU/mem ceiling (or "leave it as it is") made sameCeiling() true → skipped persistCeiling → the phantom GPU stayed. It is NOT inert: client-runtime _gpu_available_from_env treats any non-empty value as a GPU cluster (only an explicit-empty value means no GPU), so jobs-manager requests a nonexistent nvidia.com/gpu — unschedulable, forcing a GPU→CPU fallback with a false GPU heartbeat. Fix: capture phantomGPU (env has GPU but machine doesn't) before normalizing HasGPU away, and force the persist on an otherwise-no-op so BuildEnvSpec's explicit-empty override lands and clears it. New regression test TestSet_PhantomGPUForcesPersistOnNoOpCeiling (mutation-proven: fails if the gate is removed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
…om-GPU cleanup (Bugbot #241) Follow-up to 4aa7934: the phantom-GPU cleanup fell through to validateDesired on an UNCHANGED ceiling, so a machine that shrank under its current ceiling blocked the cleanup with exit 2 ("phantom GPU cleanup blocked by fit-check"). Gate validateDesired on the ceiling actually CHANGING — an unchanged ceiling mutates nothing the fit-check protects (the established no-op-when-shrunk rule), and the phantom cleanup only REMOVES a GPU request. New regression test TestSet_PhantomGPUCleanupNotBlockedByFitCheck (mutation-proven). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
…oose an amount (Bugbot #241) Bugbot MED: on a node too small to give a run even the 1-core / 2-GiB minimum after tracebloc's ~1-core, ~3-GiB overhead, the wizard's "Choose an amount" path prompted an impossible bounded range (e.g. "1–0"), so every answer was rejected and the guided flow could only be escaped by interrupting. Guard the branch: fail with an honest exit-2 "machine too small" message instead of trapping the user. New test TestWizard_ChooseAnAmountTooSmallMachine (mutation-proven). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
aptracebloc
left a comment
There was a problem hiding this comment.
Re-reviewed after the phantom-GPU fixes + develop reconciliation — the concern we held on is resolved, and I traced it end-to-end:
- Phantom-GPU (the held item): the decisive runtime question is answered (grounded in
client-runtime@develop's_gpu_available_from_env: a non-emptyGPU_REQUESTSon a GPU-less node → unschedulable pod + falsely-advertised GPU — the phantom is not inert).applyResourcesSetnow capturesphantomGPU := current.HasGPU && !machineHasGPUbefore normalizing, and forces the persist on an otherwise-no-op soBuildEnvSpec's explicit-empty override clears the stale chart default.TestSet_PhantomGPUForcesPersistOnNoOpCeilingis mutation-proven. - Follow-on fixes verified: the fit-check is correctly guarded (
if !ceilingUnchanged { validateDesired(...) }), so the cleanup can't be blocked by exit 2 on a machine that shrank; and the too-small-machine wizard now fails honestly. - All 7 Bugbot threads resolved, reconciled with develop (no conflict), CI green.
LGTM — approving.
— drafted with Claude (Opus 4.8), sent by @aptracebloc
Uh oh!
There was an error while loading. Please reload this page.
LukasWodka
commented
Jul 13, 2026
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 10f6ce8. Configure here.
…nes (#275) The chart stamps GPU_REQUESTS/GPU_LIMITS=nvidia.com/gpu=1 as literal container env on every install, even CPU-only hosts, so ParseTraining reports a phantom HasGPU=true. `resources show` then printed "per training run: up to N CPU · M GiB · 1 GPU" on a machine with no GPU — and under --verbose ALSO printed "gpu: none detected", contradicting itself. Normalize train.HasGPU to false when the node read succeeded and exposes no GPU (the same guard already used for the "none detected" detail), mirroring the set path's phantom-GPU handling (Bugbot #241). A node-read failure leaves it as-is, since we can't confirm absence. Adds a regression test. Pre-empts the Cursor Bugbot "phantom GPU" class of finding on #266. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…elease policy (#296) Every claim re-verified against the actual command tree (go run ./cmd/tracebloc --help + each subcommand on develop @ 5af9a80): README: - v0.3.0 status blurb -> v0.8.0 latest release + a lifecycle command table (login/logout/auth, client/status, resources show+set, data ingest/list/delete/validate, cluster info, doctor, delete, version/completion/home screen); notes what landed post-v0.8.0 - 15-of-16 tasks -> all 16 (semantic_segmentation shipped in #247; registry verified: 16x CLISupported, zero false) - roadmap tail: 9-of-10-modalities + v0.2/v0.3 story -> v0.2-v0.8 arc; semseg removed from Next (done); cloud-source kept (backed by RFC-0002 + the in-help 1 GiB cap copy) - links the navigation map docs/cli-navigation.md: - stale-basis disclaimer removed (was develop @ 27c5392) - resources set (#241) + status-aware home screen (#244) un-dashed — both merged; proposed classDef dropped - nonexistent home-screen-spec.md references removed (point at #244) - doctor node: top-level doctor shipped, cluster doctor = hidden alias (pinned by TestClusterCmd_DoctorIsHiddenAlias) - known-gaps list pruned to what is still true (offboard partial teardown still exits 0 — re-verified in delete.go) scripts/RELEASE_CHECKLIST.md: - 5-line release policy: trigger = customer-visible merge or weekly, owner = DevEx squad (role, not person) - EKS smoke: dataset push -> data ingest; v0.1-ticket pre-flight genericized release.yml: header comment claimed release notes come from CHANGELOG.md — no such file exists; it's generate_release_notes: true. Fixes#296 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…release policy (#307) * docs: truth pass — README/nav-map/checklist match the v0.8 surface; release policy (#296) Every claim re-verified against the actual command tree (go run ./cmd/tracebloc --help + each subcommand on develop @ 5af9a80): README: - v0.3.0 status blurb -> v0.8.0 latest release + a lifecycle command table (login/logout/auth, client/status, resources show+set, data ingest/list/delete/validate, cluster info, doctor, delete, version/completion/home screen); notes what landed post-v0.8.0 - 15-of-16 tasks -> all 16 (semantic_segmentation shipped in #247; registry verified: 16x CLISupported, zero false) - roadmap tail: 9-of-10-modalities + v0.2/v0.3 story -> v0.2-v0.8 arc; semseg removed from Next (done); cloud-source kept (backed by RFC-0002 + the in-help 1 GiB cap copy) - links the navigation map docs/cli-navigation.md: - stale-basis disclaimer removed (was develop @ 27c5392) - resources set (#241) + status-aware home screen (#244) un-dashed — both merged; proposed classDef dropped - nonexistent home-screen-spec.md references removed (point at #244) - doctor node: top-level doctor shipped, cluster doctor = hidden alias (pinned by TestClusterCmd_DoctorIsHiddenAlias) - known-gaps list pruned to what is still true (offboard partial teardown still exits 0 — re-verified in delete.go) scripts/RELEASE_CHECKLIST.md: - 5-line release policy: trigger = customer-visible merge or weekly, owner = DevEx squad (role, not person) - EKS smoke: dataset push -> data ingest; v0.1-ticket pre-flight genericized release.yml: header comment claimed release notes come from CHANGELOG.md — no such file exists; it's generate_release_notes: true. Fixes#296 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: refresh README latest-release refs v0.8.0 -> v0.9.1 (#296) Addresses Asad's review on #307: this truth pass predated the v0.9.0 and v0.9.1 cuts, so a few version claims went stale. - Status blurb: "v0.8.0 is the latest release" -> "v0.9.1 is the latest release". - The post-v0.8.0 "ships with the next release" framing for resources / status-aware home screen / top-level doctor / semantic_segmentation -> "Shipped in v0.9.0" (they released in v0.9.0, no longer "next"). - Roadmap tail: "Since the v0.8.0 cut, develop gained ..." -> a "v0.9 added ...; v0.9.1 is the current latest" entry, extending the v0.2-v0.8 arc. Historical v0.8.0 references (the v0.8.0 cut, cluster doctor in v0.8.0) kept as deliberate contrasts. Homebrew tap stays dropped (#300) - no reintroduction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
… (cli#546) (#547) Bugbot MEDIUM on the develop->staging promotion mirror (#540), against #539 (backend#2220). #539 made any non-`user` RESOURCE_PROVENANCE stale, so an unchanged ceiling stopped returning early and instead fell through to the apply in order to re-stamp the marker as `user`. Correct intent, wrong landing site: the fall-through lands in the confirmation gate, and off a terminal that gate does not ask -- it returns exit 1. So tracebloc resources set --cores 4 --memory 16 # the CURRENT ceiling went from the exit-0 no-op the command's own --help documents ("0 applied (or nothing to change)"), and that docs/cli-navigation.md draws as an edge going straight to exit 0 bypassing CONF, to a hard failure. Nearly every installed edge still reads `installer` or `unknown`, so the blast radius was the installed base rather than an edge case, and the callers that restate a size are scripts -- the bootstrap, the end-to-end journey -- none of which pass --yes for what the docs promise is a no-op. The fix is one clause on the gate, not a revert: the gate guards the CEILING, and an unchanged ceiling has nothing to ask about ("Let each training run use up to 4 CPU / 16 GiB?" when the answer is already that is a question with one honest answer). The re-stamp #539 added still happens. The phantom-GPU fall-through (#241) had the same shape and is fixed by the same clause: both are bookkeeping writes, already announced by their own Infof lines, not budget changes an operator needs to sanction. Tests: TestSet_SameCeilingNeedsNoYes. Every same-ceiling case already in the file passed `yes: true` -- the flag under dispute -- which is why nothing was red. The new cases assert exit 0 AND that the apply still happens, because either alone is satisfiable by the wrong fix; they read RESOURCE_PROVENANCE=user off the values file helm was actually handed (the existing assertions go through --dry-run, which skips the very gate at issue); they cover the phantom-GPU sibling and a declining prompter on a terminal; and one sub-case bounds the fix by proving a real CHANGE off a terminal still exits 1 and mutates nothing. Five of the six sub-cases fail without the one-clause change. docs/cli-navigation.md: the `no change` node now says it may still re-stamp provenance or clear a phantom GPU, and never asks to confirm -- true of #539's behaviour too, which shipped without updating the map. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…ne can give (backend#2221) (#571) * test(resources): pin that the wizard never offers more than the machine can give (backend#2221) #2221's fourth scope item -- "never offer a ladder rung the VM cannot honour" -- was covered by two examples (a shrunk machine, a too-small machine). The property it actually asks for is universal, and a future rung ladder (the epic's XS 4 / S 16 / M 32 / L 64 / XL 128 GiB decision of record) is exactly the change that satisfies both examples and breaks the property: offer a fixed rung and it is unhonourable by construction on a small VM. Swept over six machine shapes including #2221's own measurement (a 6 CPU / 11.67 GiB Docker Desktop VM). A machine that cannot seat the floor must fail honestly rather than offer something unhonourable, which folds in the Bugbot #241 rule as part of the same invariant. My first version was VACUOUS: it answered the prompt with a prefix that matched no option, so the wizard left the budget alone and the assertion read the CURRENT value. It passed on every machine and survived a mutation making the wizard offer a fixed 8-core / 32-GiB rung -- the precise case it exists to catch. Fixed by answering the full option string and choosing a current budget a no-op cannot hide behind; three mutations now redden it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(resources): make the no-op backstop actually back something up (backend#2221) Cursor Bugbot Medium on #571. I claimed a current budget of cpu=1,memory=3Gi made a silent no-op fail the floor check. It does not -- 1 core / 3 GiB sits at and above the 1-core / 2-GiB floor, and every shape in the table can honour it, so a wizard that never offered anything passed every assertion. The real backstop is an equality check: "use as much as possible" has an exact contract, so a no-op (which applies the CURRENT budget), a fixed rung, and a prompt string that matches no option all land somewhere other than the machine maximum and fail. A bound alone is satisfied by offering nothing. Proof it now works: re-introducing the prefix-only prompt answer -- the vacuous version this test shipped with -- reddens 5 of 6 shapes, where before it passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: shujaat hasan <shujaathasan@shujaats-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
What
P2 —
tracebloc resources set: the guided knob for how much of the machine a training run may use. Closes#143.UX — wizard by default (built for non-technical users)
tracebloc resources seton a terminal: shows the current per-run budget vs the machine (CPU · Memory · GPU — the GPU row is omitted if there's no GPU) → a pick list with "Use as much as possible" pre-selected, "Choose an amount" (prompts bounded to the machine, so over-asking is impossible), or "Leave it as it is" → plain-language confirm → "applies to your next run."--cores N(--cpuhidden alias) ·--memory SIZE(acceptsGB/Gi/bare → echoed as GiB) ·--gpus G·set max, with a full validation matrix (too-high / too-low / wrong-unit / bare-number / floor / no-op / one-dimension / off-TTY) that always gives a human message + the fix.Decision A
The number the user sets = what one training run may use (written verbatim to
RESOURCE_*). tracebloc's ~1 CPU / 3 GiB overhead is an invisible fit-check margin, never subtracted from the user's number. SHOW is unchanged.Safe live-cluster apply
helm upgrade --reset-then-reuse-values -f <temp values>(temp file dodges the comma footgun), pinned to the current chart version — refuses to run unpinned so it can never silently bump the chart; resolved--namespace/--kube-context;--wait. Landsenv.RESOURCE_*(+GPU) in the release Config, which the hourly auto-upgrade preserves (CI-proven).--dry-runmutates nothing; an unchanged value is a no-op; running training is not interrupted (only the small jobs-manager rolls; the new limits apply to the next run). macOS shows an honest "raise Docker Desktop → Resources" message rather than auto-restarting the VM (P3 deferred).GPU (first-class)
Whole units, omitted when the machine has none, no overhead math. Removing the GPU writes an explicit empty
GPU_LIMITS/GPU_REQUESTS(not omitted, notnvidia.com/gpu=0) — grounded in client-runtime's_gpu_available_from_env, where any non-empty value (even=0) reads as GPU-present, so only""means none — so--reset-then-reuse-valuescan't silently re-inherit the old GPU and the success echo is honest.Tests / gates
Wizard (preselected-max, bounded prompts, GPU-row-omitted), the flag matrix, fit-check (incl GPU + largest-single-node multi-node rule), apply (fake Runner asserts reset-then-reuse,
-f,--versionpin, resolved ctx/ns, dry-run runs nothing), no-op, GPU-removal, unpinned-refusal — mutation-proven.make cigreen (helm 83.8%, resources 96.7%, cli 75.0%).Built + adversarially reviewed; the 3 live-cluster should-fixes it surfaced (unpinned upgrade, GPU-removal no-op + dishonest echo, values-shape) are all fixed and mutation-proven.
Closes#143.
🤖 Generated with Claude Code
Note
Medium Risk
The command mutates customer clusters through
helm upgrade, but fit-checks, confirmation/--yes, version pinning, and dry-run reduce the chance of bad applies or silent chart bumps.Overview
Replaces the P2 “coming soon” stub with a working
tracebloc resources setthat raises the per-training-run CPU/memory/GPU ceiling and persists it to the client Helm release.On a TTY, users get a guided wizard (max / choose amount / leave as-is); scripts use
--cores,--memory,--gpus,max,--yes, and--dry-run. User-facing numbers are written verbatim toRESOURCE_*; ~1 core / 3 GiB is only a fit-check margin against the largest single Ready node. Changes go throughhelm upgrade --reset-then-reuse-valueswith a temp values file (newinternal/helm), chart version pinned (refuses unpinned remote upgrades), and explicit emptyGPU_*when clearing GPU or chart-default “phantom” GPU on CPU-only hosts.resourcesSHOW copy now points at the realsetcommand; tests cover validation, wizard, no-op, phantom GPU, and unpinned refusal.Reviewed by Cursor Bugbot for commit 10f6ce8. Bugbot is set up for automated code reviews on this repo. Configure here.