feat(envelope): record WHO chose the training envelope (backend#2220)

[shujaatTracebloc]

Follows #766 (merged as 3c40ef9). Rebased onto develop, so this diff is provenance only.

What

Scope bullet 4 of backend#2220 — "migration for edges already pinned by resources set: they must not be stranded."

The blocker was never the migration mechanics. It was that provenance is currently unknowable — which Lukas flagged as "the actual hard part of this ticket, not the consolidation." This adds the marker that makes it knowable going forward, and is honest about the edges where it never will be.

Why it can't be inferred

RESOURCE_* has no unset state once Helm has seen it — the fleet auto-upgrade CronJob, the installer reconcile path, and the CLI's own apply all use --reset-then-reuse-values, which re-applies stored user-supplied values indefinitely.

_existing_training_resources carries any value forward unless it equals the historic cpu=2,memory=8Gi, and that is the only discriminator available. So an installer-written envelope and a deliberate tracebloc resources set are indistinguishable. Without a marker, future automatic sizing must either strand every already-pinned edge, or silently overrule operators who chose a size on purpose.

env.RESOURCE_PROVENANCE

Value	Meaning
installer	sized to this machine at install time
user	explicit resources set, or a TRACEBLOC_TRAINING_RESOURCES override
unknown	carried forward from before this key existed

unknown is a deliberate answer, not a placeholder. Guessing installer there would risk overruling a human, so consumers must treat unknown as user. Existing edges will report unknown and keep their current size until someone opts in explicitly — offering that opt-in is a product decision, not one P0 should make silently.

An existing marker is authoritative and survives re-install, in both installers. Without that, a re-install would quietly downgrade a user choice to unknown, and the marker would be worse than useless — it would decay toward the very ambiguity it exists to remove. Pinned in both bats and Pester.

The un-pin recipe, verified rather than assumed

Trap 2 in the issue's handover: "removing the key from the chart does not remove it from existing releases… the migration needs an explicit helm-side unset."

Helm deletes null-valued keys during value coalescing, so this drops all three vars and makes the derive path reachable — confirmed by rendering the chart both ways (6RESOURCE_* occurrences when set, 0 with null):

helm upgrade "$NAMESPACE" tracebloc/tracebloc -n "$NAMESPACE" \
--reset-then-reuse-values \
--set env.RESOURCE_LIMITS=null \
--set env.RESOURCE_REQUESTS=null \
--set env.RESOURCE_PROVENANCE=null

No chart change needed. MIGRATION.md carries it with a warning that matters: derivation is gated off (DERIVE_JOB_ENVELOPE, backend#2167), so clearing the keys today returns the edge to the fixed cpu=2,memory=8Gi literal — which on a sub-8GiB machine cannot schedule at all.

Implementation notes

• _training_resources is now a thin wrapper over _resolve_training_size, which returns the size and the verdict from one pass. Deriving them separately would mean either running the bounded cluster probes twice, or re-implementing the branch logic beside itself — the exact duplication this ticket deletes. All 15 pre-existing size tests pass unmodified.
• The marker rides inside the same template gate as the pair it describes: a provenance marker with no envelope to describe is noise, and gating them together means one explicit unset clears all three.
• Excluded from both generic env passthrough loops — otherwise it renders twice per container.

Verification

• bats 172 ok, 8 new provenance tests. The 3 failures (ci: add advance-deploy-env caller workflow #64–66) are pre-existing on clean develop — re-confirmed by stashing; they need real helm
• Pester 750 passed / 0 failed, 7 new
• helm unittest 480 passed vs 477 on develop — 3 added, and no new failures among the 5 pre-existing ones (unrelated suites, schema-spec errors present on clean develop)
• helm lint clean; PSScriptAnalyzer 0 errors; gen-envelope-embed.sh --check clean
• Chart 1.9.48 → 1.9.49, version + appVersion in lockstep; manifest.sha256 regenerated (both installers moved)

What this does not do

It doesn't un-strand anything by itself, and it can't. Silent un-strand­ing is unsafe for exactly the reason above — for unknown edges we do not know whose choice we'd be overriding. What this buys is that from 1.9.49 the answer is recorded, so the ladder work behind #2167 can re-derive installer values and leave deliberate choices alone.

tracebloc resources set must also stamp user, or a user overriding an installer-marked edge would leave a stale installer marker on a deliberate human choice — the precise failure this marker exists to prevent. That's the companion cli PR.

Refs: RFC-BACKEND-664 §P0, client-runtime#358, client#766, cli#538, backend#2167

🤖 Generated with Claude Code

---

Note

^{Cursor Bugbot is generating a summary for commit 8b67114. Configure here.}

[saadqbal]

The premise is exactly right, and it's the part of backend#2220 that actually needed thinking about rather than typing. RESOURCE_* having no unset state once Helm has seen it — because the auto-upgrade CronJob, the installer reconcile path and the CLI apply all use --reset-then-reuse-values — means an installer-written envelope and a deliberate resources set are genuinely indistinguishable, and cpu=2,memory=8Gi being the only discriminator is a measurement, not a guess. Adding the marker going forward and being honest that pre-marker edges are unknown forever is the right call over inventing a heuristic.

Bugbot's Medium is real, and it's a fresh instance of the exact class #766 existed to remove. Read both sides:

install-k8s.ps1:4069-4092 — Get-TrainingProvenance performs its ownhelm get values, separate from the one Get-TrainingResources does immediately below, and the enclosing try ends in a bare catch {} falling through to:

 } catch {}
# 2. Sized to this machine, or 3. the static default -- both are OUR choice.return"installer"

The bash twin resolves both in one pass — _training_provenance() calls _resolve_training_size and reads $_TB_TRAINING_PROVENANCE, so size and provenance cannot disagree — and answers unknown when a carried size has no readable marker.

So on Windows, if the size path succeeds and carries a live RESOURCE_LIMITS but the provenance lookup then throws — a wedged API, a ConvertFrom-Json hiccup, anything the catch {} eats — the generated values pin that carried envelope as installer. A future ladder trusting that label re-derives and overrules what may well have been a human choice. unknown is the safe verdict and it's what the other installer already gives.

Two things make this worth fixing rather than noting:

• It's the two installers diverging by construction, which is the whole reason #766 exists. Same file pair, three days later, in the fail-unsafe direction.
• The bare catch {} is the standing hazard #766's own description flagged as "worth its own ticket" — "any exception on that path silently hands the customer the unschedulable literal with no diagnostic". It's now silently handing them a wrong provenance verdict too. Whether or not you widen that ticket here, this call site shouldn't fall through to the unsafe answer.

The shape that fixes both at once is the bash one: resolve size and provenance together in a single lookup, and let a failed read produce unknown rather than installer. That also drops a redundant helm get values per install.

Requesting changes on that; the Bugbot thread blocks the merge anyway.

Merge ordering, for whoever lands these: this and cli#539 are only correct together, per #539's description — the marker without the user side is worse than no marker. Note that #539 currently has its own open High (sameCeiling returns before BuildEnvSpec, so a same-size resources set keeps installer), so both need fixing before either is safe to land, not just sequencing.

[saadqbal]

Approving. Green (43 passing, 4 skipped, nothing failing or pending), no open threads, and the conflict from earlier is resolved.

My finding is fixed, and fixed at the right level.Get-CarriedTrainingValues is now one shared lookup returning @{ Size; Provenance } or $null, and I checked the consequence rather than the diff: Get-TrainingProvenance still answers installer when it gets $null, and that is now correct — because Get-TrainingResources receives the same $null from the same read and falls through to machine sizing. So installer is a true statement about a machine-sized envelope rather than a label pinned onto a carried one. The two answers can no longer disagree because there is only one answer.

The call site is what makes that hold, so I checked it too (:5023-5027):

$carried=Get-CarriedTrainingValues$trainingSize=Get-TrainingResources-Carried $carried-CarriedResolved
$trainingProvenance=Get-TrainingProvenance-Carried $carried-CarriedResolved

Resolved once, passed to both. And install-k8s.Tests.ps1:1690 pins exactly that property — "size and provenance come from ONE lookup and cannot disagree" — which is the invariant rather than a restatement of the code. That also drops a helm get values per install, and brings the ps1 into line with the bash twin's single _resolve_training_size pass, which is what I'd asked for.

Good catch on b204256 too — bumping to 1.9.50 because #767 took 1.9.49. develop is at 1.9.49 and this is at 1.9.50, so that's right. Worth noting that's the same collision I flagged on #764; git won't surface it when both sides say the same version identically, so it's easy to lose.

Merge ordering — please ship this with cli#539, not ahead of it. Both descriptions say the pair is only correct together, and that's accurate but worth stating precisely so the risk isn't over- or under-read:

• Nothing consumes the marker yet — the comment at :5031 says it's bookkeeping and never changes the envelope — so landing this alone breaks nothing today.
• But any edge the installer stamps installer and an operator then re-sizes by hand will keep saying installer until #539 is in, and that wrong value is silent and sticks. It only becomes damaging when a consumer trusts it, which is exactly what the future ladder will do.

And note cli#539 currently has its own open High — sameCeiling returns before BuildEnvSpec, so a same-size resources set doesn't stamp user either. So the pair isn't yet correct together; that one needs fixing before either lands, not just sequencing.

The premise remains the best part of this: RESOURCE_* having no unset state once Helm has seen it, with cpu=2,memory=8Gi as the only discriminator, is a measurement rather than a guess — and being honest that pre-marker edges are unknown forever beats inventing a heuristic that would be wrong quietly.

[shujaatTracebloc]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

1 issue from previous review remains unresolved.

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b34c13c. Configure here.}

[shujaatTracebloc]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 25c3ca9. Configure here.}

[saadqbal]

Re-approving. Green (43 passing, 4 skipped, nothing failing or pending), no open threads.

Scope note first, since it changed:#769 has merged into this branch, so this is no longer provenance-only — it now carries the undersized-envelope work too, which I reviewed on #769 and found sound. Chart version checked because this is the fourth time I've seen that trap today: develop is at 1.9.49 and this is at 1.9.50. Correct, no collision.

The |unschedulable fix is my #769 note, and it turned out sharper than I framed it. I'd flagged the re-probe as a minor "second lookup that can disagree with the first". The real defect was worse: _machine_training_ceiling returns nothing for four different reasons and only one is "too small", so any listable node — one whose allocatable didn't parse, or one not yet Ready — tripped a hard "this machine is too small". Your own commit message names the contradiction better than I did: the comment said "warning about machine size would be a fabrication" while the code fabricated exactly that.

The resolution is the right one — a fourth distinct return rather than a smarter inference:

|viable -> sized to the machine
|undersized -> the honest remainder, warn
|unschedulable -> literal, warn (MEASURED, genuinely too small)
(fall-through) -> literal, SILENT (never measured, so any claim would be invented)

Two states that were one are now two, and the silent arm's comment says why. That's the same shape as the provenance fix earlier in this PR, which is presumably why you called it the third instance of the class.

And the bash twin now matches the ps1 — one _resolve_training_size pass returning size and marker together. The cases pin it directly: size and marker come from ONE pass and always agree, plus a carried size with no marker pairs with 'unknown' from the SAME read. 27 cases green across training-size and provenance, and the matrix is complete — only after measuring, stays SILENT when nothing parses, a failed values read carries NOTHING (never a stranding 'unknown'), a junk marker degrades to 'unknown', never to a guess, and an existing 'user' marker SURVIVES re-install.

That last group is the part worth keeping: every arm answers "what do we actually know" rather than "what's the most likely value", which is the whole premise of the marker.

Merge ordering still applies. This and cli#539 are only correct together — I've approved both. Ship them together rather than either alone.