Uh oh!
There was an error while loading. Please reload this page.
refactor(installer): read the training envelope from the contract, stop computing it (backend#2220) - #766
Conversation
…op computing it (backend#2220) Both installers computed "how much of this machine may one training run have" from scratch, with a third copy in cli's set.go and a fourth, disagreeing policy in client-runtime's node_sizing.py. client-runtime#358 made the arithmetic single-definition; this makes the installers readers of it. Deleted from install-client-helm.sh and install-k8s.ps1: the 1-CPU/3-GiB overhead, the 1-core/2-GiB floor, and — the part nobody had chosen — the (memory, cpu) node ordering. That last one is a real divergence, not just duplication: cli's nodeLarger ranked (cpu, memory), so on a cluster of 8c/16Gi + 4c/32Gi the installer and `tracebloc resources set` anchored on DIFFERENT nodes and disagreed about the same machine. One order now, the contract's, matching cli. Installer-provisioned clusters are single-node k3d, where the orders cannot differ, so the field sees no change. Neither installer can call the Python: bash has no guaranteed JSON parser (jq is not a prerequisite — the helm-namespace parser says so in as many words) and this bootstrap is SIGNED, so it must not fetch anything unsigned at install time. So the constants are EMBEDDED and the embed is drift-guarded, the same trade the GPU node-image build inputs already make (#616/#633). Deliberately NOT a new scripts/lib/*.sh: install-k8s.ps1 is a single self-contained file and cannot source a bash lib, so a lib would have meant two mechanisms, two guard shapes, and five more sync points on the signed file surface. Three ways that can rot, one gate each: 1. the embed stops matching the vendored contract -> scripts/gen-envelope-embed.sh --check, mirrored in bats AND Pester 2. the vendored contract stops matching upstream -> envelope-contract-drift.yml, against scripts/.client-runtime-ref 3. the installers stop reproducing the contract's vectors -> the golden-vector replays, in the normal test jobs Pin, don't float (cli/.github/workflows/chart-drift.yml's rule): an unrelated client-runtime commit must not redden every open client PR; the weekly run catches a pin gone stale. The drift job mints a least-privilege App token (named repositories, contents:read) because client-runtime is private and GITHUB_TOKEN cannot read another repo. Fixes a live bug found while writing the Pester replay: [math]::Max(0, ...) binds the (Int32, Int32) overload and throws on any byte count over 2^31 ("Value was either too large or too small for an Int32"). The enclosing bare `catch {}` swallowed that into a silent fall-through to cpu=2,memory=8Gi — the first replay run had EVERY machine returning the literal. Now [long]0. Note the `catch {}` itself is a standing hazard: any exception on that path silently hands the customer the unschedulable literal with no diagnostic, and PSScriptAnalyzer has been flagging it advisory-only (PSAvoidUsingEmptyCatchBlock at :4073 and :4122). Filed separately rather than widening this PR. manifest.sha256 regenerated — both installers' digests moved. Verified: bats 164 pass (the 3 failures are pre-existing on clean develop — they need real helm, the gap client#751 just closed in CI); Pester 699/0; `make lint` clean at severity=error over 47 files; PSScriptAnalyzer 0 errors; gen-manifest/index-invariants/check-facts/leftover-guard/bats-hygiene all green. Drift detection proven by tampering a constant and repairing it. No client/** chart files touched, so no Chart.yaml bump. Refs: RFC-BACKEND-664 P0, client-runtime#358, cli#143 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
…#766) The generator grepped in one place and rewrote in another, so a single string had to serve as both an ERE pattern and a Python literal. The PowerShell prefix was passed with a leading backslash so grep would read the dollar as literal; that backslash then reached re.escape(), which searched install-k8s.ps1 for a backslash-dollar assignment that does not exist there. --check was unaffected, because it never gets past the grep. So the gate looked healthy while the documented adopt path was broken: a real regen rewrote the bash installer, died on the first PowerShell constant, and left the two embeds disagreeing about the same contract -- the exact drift this whole mechanism exists to prevent. Reproduced before fixing: bash moved to 4294967296, ps1 stayed at 3221225472, exit 1. Now python3 does the whole job -- find, compare, and rewrite -- against a LITERAL key, so there is one escaping domain instead of two. Exit codes carry the outcome back to bash (0 correct/rewritten, 2 no such assignment, 3 present but wrong under --check), which also drops the leaked backslash from the drift message. Why the existing tests missed it, which is the part worth keeping: they only ever ran --check (a read) or a regen against an already-correct contract, where every constant short-circuits before the rewrite. The mutating path had no coverage at all. Two tests added that do have it, both in a scratch copy so they cannot mutate the repo: * adopt a genuinely changed contract, assert BOTH installers moved and the result is self-consistent * delete a PowerShell constant, assert the generator fails LOUDLY rather than reporting a healthy embed -- the fail-open shape gen-manifest.sh warns about in its own empty-surface guard Verified: 6/6 envelope tests, bats-hygiene 18/18, install-client-helm 162 ok with the same 3 pre-existing failures clean develop has (#64-66, they need real helm), make lint clean over 47 files, manifest unchanged (this touches only dev tooling and tests, neither of which is on the signed file surface). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 27cad8f. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
…em (Bugbot #766) Get-TrainingResources coerced an unparseable CPU or memory quantity to 0 and then ranked the node anyway. The contract's skipped_nodes says such a node is SKIPPED, and the bash twin has always done that with an explicit `|| continue` — the ps1 was the odd one out. The old memory-first order hid it: a memB of 0 could never win the anchor. Ranking cpu-first, which this PR introduced for parity with cli, exposes it. A node with a good core count and a memory unit we do not speak takes the anchor, fails the memory floor, and drops the WHOLE machine to cpu=2,memory=8Gi while a sibling node was perfectly sizeable. So the reorder was safe on k3d but not on a heterogeneous BYO cluster. Reproduced before fixing, as a test: nodes "16 64GB" + "8 32Gi" returned cpu=2,memory=8Gi where cpu=7,memory=29Gi was correct. $null now marks unparseable and the node is skipped, matching bash and the contract. Coverage added on both sides, because the single-node golden vectors could not have caught this — they only ever feed ONE node, and the bug needs an unparseable node COMPETING with a valid one: Pester: unparseable memory / unparseable cpu / all-unparseable bats: the same two cases against _machine_training_resources, pinning that the twins agree rather than assuming it bash needed no change — its test passed on the first run, which is the point of adding it. manifest.sha256 regenerated: install-k8s.ps1's digest moved. Verified: Pester 743 passed / 0 failed / 13 skipped, envelope bats 8/8, bats-hygiene 18/18, PSScriptAnalyzer 0 errors, make lint clean over 47 files, gen-envelope-embed --check clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
saadqbal
left a comment
There was a problem hiding this comment.
Reviewed this as approve-ready and it is, with one procedural blocker: the Bugbot Medium on Get-TrainingResources is stale — 9bdaaf6b ("skip unparseable nodes in the ps1 twin, don't rank them") is the head commit and fixes exactly what the thread describes. $cpuM/$memB are now $null rather than 0, and if ($null -eq $cpuM -or $null -eq $memB) { continue } skips the node, matching _machine_training_resources and the contract's skipped_nodes. Could you resolve that thread? required_conversation_resolution is on, so it blocks the merge whatever the severity, and I can't approve over an open Bugbot finding even when the code has moved past it. I'll approve on my next pass once it's closed.
On the substance, I exercised the guards rather than reading the table:
scripts/gen-envelope-embed.sh --checkis clean on the tree, and tampering the ps1 constant is caught with the file, the wrong value, the right value and the remedy all named:That's the right thing to have proven, given the guard's own recorded failure mode was a regen that rewrote one installer whileEMBED DRIFT: scripts/install-k8s.ps1 $script:TbEnvelopeOverheadCpuMilli = 999, contract says 1000--checkstill passed — good that27cad8f5closed it and that the comment at:68records why.[long]0at:4123-4124is the real find here. A bare0binding the(Int32, Int32)overload and throwing on any byte count over 2³¹, swallowed by the enclosing barecatch {}into a silentcpu=2,memory=8Gi, is exactly the class of bug that never gets reported — the customer just gets a wrong-sized envelope and no diagnostic. Finding it because every machine returned the literal in a Pester replay is a good argument for the replays existing.
Two things I agree with rather than would change:
- Embedded constants over a lib file.
install-k8s.ps1being self-contained means a shared lib buys you two mechanisms and five more sync points on a signed surface, and the bootstrap must not fetch anything unsigned. The drift gate is the right price to pay instead. - Pin, don't float on
.client-runtime-ref, plus re-deriving upstream's vectors from upstream's code rather than trusting the vendored file. A contract that matches upstream byte-for-byte while upstream's own goldens have gone stale is a genuinely nasty failure mode and it's worth the extra step.
The (memory, cpu) → (cpu, memory) reorder is the part I'd want a second pair of eyes on at some point, but the reasoning holds: it was an unchosen divergence from cli's nodeLarger, the two anchored on different nodes for the same heterogeneous cluster, and installer-provisioned clusters are single-node k3d so the field sees nothing. Fine to land as a no-op that removes a real disagreement.
Also noting the standing catch {} hazard is filed separately rather than widened into here — right call for a PR this size.

What
Step 2 of 3 for
backend#2220/ RFC-BACKEND-664 §P0. client-runtime#358 (merged) made the training-envelope arithmetic single-definition; this makes both installers readers of it instead of producers.Deleted from
install-client-helm.shandinstall-k8s.ps1:(memory, cpu)node orderingThat last one is a real divergence, not just duplication.
cli'snodeLargerranked nodes(cpu, memory), so on a cluster of 8c/16Gi + 4c/32Gi the installer andtracebloc resources setanchored on different nodes and disagreed about the same machine. Nobody chose that — it fell out of two independent implementations. One order now, the contract's, matchingcli. Installer-provisioned clusters are single-node k3d where the orders cannot differ, so the field sees no change.Why embedded constants and not a lib file
Neither installer can call the Python. bash has no guaranteed JSON parser — jq is not a prerequisite, as the helm-namespace parser at
:204says in as many words — and this bootstrap is signed, so it must not fetch anything unsigned at install time. So the constants are embedded and the embed is drift-guarded: the same trade the GPU node-image build inputs already make (#616/#633).Deliberately not a new
scripts/lib/envelope-contract.sh:install-k8s.ps1is a single self-contained file and cannot source a bash lib, so a lib would have meant two mechanisms, two guard shapes, and five more sync points on the signed file surface (install.shFILES,gen-manifest.shFILES,install.ps1$Files, WINDOWS_FILES, plus the manifest).Three ways this can rot, one gate each
scripts/gen-envelope-embed.sh --check, mirrored in both bats and Pesterenvelope-contract-drift.yml, againstscripts/.client-runtime-refPin, don't float —
cli/.github/workflows/chart-drift.yml's stated rule: an unrelated client-runtime commit must not redden every open client PR, and the weekly run catches a pin gone stale enough to matter. Pinned toclient-runtime@6293d15(the #358 merge on develop), verified byte-identical to the vendored fixture.The drift job mints a least-privilege App installation token (
repositories: client-runtime,permission-contents: read) because client-runtime is private andGITHUB_TOKENcannot read another repo — unlike cli's chart-drift, which reads the publictracebloc/clientwith no token. Scoping follows the backend#2157 sweep.It also re-derives upstream's vectors from upstream's own code, rather than trusting the file: a vendored contract can match upstream byte-for-byte while upstream's goldens have gone stale against its own arithmetic, and a stale contract reaching us is our problem.
A live bug this turned up
[math]::Max(0, $bestMemB - $overhead)binds theInt32overload and throws on any byte count over 2³¹ — "Value was either too large or too small for an Int32". The enclosing barecatch {}swallowed it into a silent fall-through tocpu=2,memory=8Gi. My first Pester replay run had every machine returning the literal, which is how it surfaced. Now[long]0.The
catch {}itself is a standing hazard worth its own ticket: any exception on that path silently hands the customer the unschedulable literal with no diagnostic, and PSScriptAnalyzer has been flagging it advisory-only (PSAvoidUsingEmptyCatchBlockat:4073and:4122). Filed separately rather than widened into here.Verification
develop— verified by stashing; they need real helm behaviour, the same gapclient#751just closed in CIGet-TrainingResourcestestsmake lint(required): 41 scripts parse, shellcheckseverity=errorclean over 47 filesgen-manifest12/12,index-invariants17/17,check-facts14/14,leftover-guard32/32,bats-hygienegreen--checknamed the file, the value and the fix; regenerating repaired itmanifest.sha256regenerated — both installers' digests movedclient/**chart files touched, so noChart.yamlbumpNext
set.gokeeps its Decision-A contract verbatim (the number the user sets IS the ceiling, written unchanged), drops theMaxRunCores/MaxRunGiBarithmetic, adds the pinned-ref drift gate, and carries the amendment comment ontocli#143.cpu=2,memory=8Gi. Needs a sibling key plus an explicit helm-side unset before un-stranding can be safe.cpu=2,memory=8Gi, which on a 4 GiB box is larger than the box. The accessor now reportsviable=Falsefor exactly this case; wiring the installers to refuse honestly is a behavior change and lands on its own so it stays revertable.Refs: RFC-BACKEND-664 §P0, client-runtime#358, cli#143, #616/#633
🤖 Generated with Claude Code
Note
Cursor Bugbot is generating a summary for commit cd75433. Configure here.