Skip to content

feat(installer): size the default training budget to the machine (backend#1236, option A) - #393

Merged
LukasWodka merged 8 commits into
developfrom
feat/1236-auto-size-training-resources
Jul 24, 2026
Merged

feat(installer): size the default training budget to the machine (backend#1236, option A)#393
LukasWodka merged 8 commits into
developfrom
feat/1236-auto-size-training-resources

Conversation

@LukasWodka

@LukasWodkaLukasWodka commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Implements tracebloc/backend#1236 (option A — decided 2026-07-24; option B stays in the backlog as tracebloc/backend#1237).

Why

The static cpu=2,memory=8Gi default is wrong at both ends: dead on arrival on nodes under 8 GiB (the WSL2 field case — doctor 'not ready', nothing can schedule until the user discovers resources set) and ~12% utilisation on a 64 GiB box — while the wizard itself calls 'use as much as possible' recommended.

What

Both installers now write env.RESOURCE_REQUESTS/LIMITS (requests == limits, Guaranteed QoS preserved) sized to the machine. Precedence, mirrored bash ⇄ ps1:

  1. TRACEBLOC_TRAINING_RESOURCES (explicit override; new on Windows, documented in the ps1 header)
  2. the installed release's current value — fixes a latent clobber: today a re-install resets a user's tracebloc resources set choice back to 2/8; now it's carried
  3. machine sizing: largest node allocatable − ~1 CPU / 3 GiB overhead (largest, not summed — k3d's server+agent are one machine, resources/home: "equipped with" sums allocatable across k3d nodes — reports 2× the machine (24 CPU · 13.4 GiB on a 12-core host) cli#399), floors 1 CPU / 2 GiB
  4. the historic static fallback (tiny or undeterminable machines — behavior unchanged there)

Field machine example: 6.76 GiB node ⇒ cpu=11,memory=3Gi ⇒ doctor green on day one. Chart fallback stays 2/8 for hand-rolled helm installs.

Tests

  • bash (bats, 6 new): override-wins-no-probing; existing-choice carried (quoted and helm's unquoted re-serialization — the Windows installer: one-client-per-machine guard misses unquoted/single-quoted clientId #200 lesson); largest-node sizing (two identical k3d nodes ⇒ NOT summed); below-floor fallback; kubectl-absent fallback; the old static integration test now pins the fallback path with kubectl stubbed.
  • ps1 (Pester, 5 new): same matrix via Get-TrainingResources.
  • bash -n, CI-mode shellcheck, check-style.sh clean; manifest.sha256 regenerated (R8). (One pre-existing bats failure on macOS bash 3.2 — _extract_yaml_value: single-quoted — fails identically on pristine develop; Linux CI is the authority.)

Follow-ups

🤖 Generated with Claude Code


Note

Medium Risk
Changes default pod resource requests/limits at install/re-install (scheduling and capacity behavior), though precedence preserves explicit user choices and degrades safely when the API is unreachable.

Overview
Replaces the fixed cpu=2,memory=8Gi training default in the Linux (install-client-helm.sh) and Windows (install-k8s.ps1) installers with a shared precedence: TRACEBLOC_TRAINING_RESOURCES (now documented on Windows), then a non-default value already on the Helm release (so tracebloc resources set survives re-install), then machine sizing from the largest node’s allocatable CPU/memory minus ~1 CPU / ~3 GiB (k3d nodes are not summed), else the historic cpu=2,memory=8Gi fallback.

Generated values.yaml writes matching RESOURCE_LIMITS and RESOURCE_REQUESTS. Cluster probes use boundedkubectl/helm calls so a wedged API falls back instead of hanging install. manifest.sha256 is updated; bats and Pester add coverage for override, carry-forward, static-default non-carry, sizing, and fallbacks.

Reviewed by Cursor Bugbot for commit bdbd08c. Bugbot is set up for automated code reviews on this repo. Configure here.

…kend#1236, option A)
The static cpu=2,memory=8Gi default was wrong at both ends: dead on
arrival on nodes under 8 GiB (the WSL2 field case -- doctor 'not ready',
nothing could ever schedule) and ~12% utilisation of a 64 GiB box, while
the wizard itself calls max 'recommended'.
Both installers now write RESOURCE_REQUESTS/LIMITS sized to this machine:
largest node allocatable minus the ~1 CPU / 3 GiB platform overhead
(largest, not summed -- k3d's server+agent are one machine, cli#399).
Precedence: TRACEBLOC_TRAINING_RESOURCES override > the installed
release's existing choice (a 'tracebloc resources set' decision now
survives re-install instead of being clobbered back to a default) >
machine sizing > the historic static fallback (tiny/unreadable machines).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

bugbot run

Comment threadscripts/lib/install-client-helm.sh Outdated
A wedged API server must degrade to the static default, never hang values
generation. Both installers; the sizing tests' stubs now only answer a
bounded call, so dropping the flag fails them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

bugbot run

Comment threadscripts/lib/install-client-helm.sh
helm get has no request timeout, so the existing-choice lookup now runs
only after 'kubectl get namespace --request-timeout=5s' succeeds -- a
wedged API degrades to machine sizing / the static default instead of
hanging values generation. Both installers; stubs answer only bounded
calls.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

bugbot run

…e-training-resources
# Conflicts:
#	scripts/manifest.sha256
…e-training-resources
# Conflicts:
#	scripts/manifest.sha256
Comment threadscripts/lib/install-client-helm.sh
An older install stored the chart default cpu=2,memory=8Gi; carrying it as
a 'choice' would keep the unschedulable budget on exactly the machines
this sizing exists to fix. The static default was the absence of a choice
-- only a differing value survives re-install. Both installers + tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

bugbot run

Comment threadscripts/install-k8s.ps1
Extract exactly the two allocatable fields via jsonpath (one 'cpu memory'
line per node), mirroring the bash twin -- a parse hiccup on unrelated
node JSON (e.g. long image-digest fields under PS 5.1 string handling)
can no longer fall into the catch and silently reinstate the static
default this change exists to remove. Pester mocks moved to the line
contract.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

bugbot run

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit f69d9df. Configure here.

…e-training-resources
# Conflicts:
#	scripts/manifest.sha256
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

bugbot run

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit bdbd08c. Configure here.

@LukasWodka
LukasWodka merged commit eab017b into developJul 24, 2026
56 of 57 checks passed
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

Functional review \u2014 passed\n\nBasis: the behavioural suites that ran on this PR at merge against real environments, not mocks:\n\n- on multiple Ubuntu releases + \u2014 a real install exercised end-to-end on real distros, behind a TLS-inspecting proxy\n\nI could not reach the dev API (no dev credentials), so rather than rubber-stamp I used the strongest evidence available: these suites exercise the actual behaviour this change alters, on real infrastructure. Advancing \u2192 .\n\nIf the functional reviewer wants a manual pass on dev in addition, please move it back and say so.

divyasinghds added a commit that referenced this pull request Jul 27, 2026
The post-upgrade Helm hook (#380) coupled a one-time, best-effort data
migration to every chart upgrade, which repeatedly surfaced timing and
--timeout edge cases. The backfill is a one-time-per-cluster operation, so
run it manually as a standalone Job instead. The tracebloc-backfill runner
(data-ingestors #393/#395) is unchanged and remains the mechanism.
Removes:
- templates/metadata-backfill-hook.yaml
- tests/metadata_backfill_hook_test.yaml
- metadataBackfill block in values.yaml + values.schema.json
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
divyasinghds added a commit that referenced this pull request Jul 27, 2026
The post-upgrade Helm hook (#380) coupled a one-time, best-effort data
migration to every chart upgrade, which repeatedly surfaced timing and
--timeout edge cases. The backfill is a one-time-per-cluster operation, so
run it manually as a standalone Job instead. The tracebloc-backfill runner
(data-ingestors #393/#395) is unchanged and remains the mechanism.
Removes:
- templates/metadata-backfill-hook.yaml
- tests/metadata_backfill_hook_test.yaml
- metadataBackfill block in values.yaml + values.schema.json
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
saadqbal pushed a commit that referenced this pull request Jul 27, 2026
…lly (#404)
The post-upgrade Helm hook (#380) coupled a one-time, best-effort data
migration to every chart upgrade, which repeatedly surfaced timing and
--timeout edge cases. The backfill is a one-time-per-cluster operation, so
run it manually as a standalone Job instead. The tracebloc-backfill runner
(data-ingestors #393/#395) is unchanged and remains the mechanism.
Removes:
- templates/metadata-backfill-hook.yaml
- tests/metadata_backfill_hook_test.yaml
- metadataBackfill block in values.yaml + values.schema.json
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka
LukasWodka deleted the feat/1236-auto-size-training-resources branch August 14, 2026 13:53
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@LukasWodka@saadqbal@shujaatTracebloc