Uh oh!
There was an error while loading. Please reload this page.
fix(#459): isolate e2e path 2 from path 1's --reuse-values replay contamination - #529
Merged
Merged
Conversation
…tamination `helm upgrade --reuse-values` records the previous release's COMPUTED values (chart defaults + overrides) as the new release's user-supplied values. Every later --reset-then-reuse-values (the fleet auto-upgrade) then replays those frozen defaults as if an operator set them, so an edge ever hand-upgraded with --reuse-values silently stops receiving chart-default updates — most importantly images.ingestor.prodDigest. e2e-auto-upgrade.sh modelled this on ONE release: path 1 (--reuse-values) contaminated the recorded values, then path 2 (the fleet auto-upgrade) ran on that contaminated release. Path 2's pin assertion passed only because the baseline pin and the working-tree pin coincided — the ERA NOTE tripwire: the first prodDigest bump would trip it. Resolution (issue decision 1 — isolate path 2 from path 1): between the paths, reset the release's recorded values to just the genuine install-time overrides (helm upgrade --reset-values --set clientId/clientPassword/storageClass), so paths 2-4 assert CLEAN-edge auto-upgrade behavior — the fleet's real contract on an edge no one hand-upgraded. Added an assertion using the issue's own contamination fingerprint (`helm get values` WITHOUT --all must not carry the chart-default prodDigest key), and rewrote the ERA NOTE to document the resolution. A contaminated REAL edge stays a separate fleet-audit concern; its remediation is exactly this reset. shellcheck + check-style clean; bash -n ok. (Runs in the label-gated e2e job — needs a real k3d cluster + the published chart, so it's not exercised by default PR CI.) Closes#459 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
shujaatTracebloc
marked this pull request as ready for review
August 3, 2026 06:56
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 301c1f8. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
… a real test (Bugbot) The isolation reset targeted $CHART_DIR (the local working-tree chart), so it pre-applied the local chart's new defaults (working-tree prod pin, egress gateway) BEFORE path 2. Path 2 then became a same-version no-op whose live prodDigest / "new defaults flowed" assertions already held from the isolation step — a --reset-then-reuse-values → --reuse-values regression would slip through (computed values would still carry the local pin). Reset to the PUBLISHED chart ($PREV) instead, so path 2 is a genuine published→local upgrade that MUST pull the new defaults for its assertions to pass. shellcheck + check-style + bash -n clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
saadqbal
approved these changes
Aug 3, 2026
Uh oh!
There was an error while loading. Please reload this page.
LukasWodka
commented
Aug 3, 2026
Contributor
/fr-pass |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

#459 — chart-default pin updates never reach an edge once hand-upgraded with
--reuse-valueshelm upgrade --reuse-valuesrecords the previous release's computed values (chart defaults + overrides, materialized) as the new release's user-supplied values. From then on, every--reset-then-reuse-valuesupgrade — exactly what the fleet auto-upgrade CronJob runs — replays those frozen defaults as if an operator set them, winning over the new chart's defaults. So an edge ever hand-upgraded with--reuse-valuessilently stops receiving chart-default updates — most importantlyimages.ingestor.prodDigest.Where it bit: the e2e tripwire
e2e-auto-upgrade.shmodelled this on one release: path 1 (--reuse-values) contaminated the recorded values, then path 2 (the fleet auto-upgrade) ran on that same contaminated release. Path 2's pin assertion passed only because the baseline pin and the working-tree pin coincided — the ERA NOTE tripwire the issue documents: the firstprodDigestbump would trip it.Fix (issue decision 1 — isolate path 2 from path 1)
Between the paths, reset the recorded values to just the genuine install-time overrides:
so paths 2–4 assert clean-edge auto-upgrade behavior — the fleet's real contract on an edge no one hand-upgraded. The working-tree pin now lands on path 2 regardless of the baseline era; no false trip on a future bump.
Added an assertion using the issue's own contamination fingerprint —
helm get valueswithout--allmust not carry the chart-defaultprodDigestkey after the reset — and rewrote the ERA NOTE to document the resolution.A contaminated real edge (one an operator hand-upgraded with
--reuse-values) stays a separate fleet-audit concern; its remediation is exactly this reset (--reset-values+ re-apply intended overrides), noted in-code.Acceptance criteria
images.ingestor.prodDigestbump flows onto the installed edge on path 2 instead of tripping the tripwire.Gates
shellcheck --severity=error,check-style,bash -nclean.e2e-auto-upgrade.shruns in the label-gated e2e job (real k3d cluster + published chart), so it isn't exercised by default PR CI — the change is shellcheck-clean and follows the script's existing idioms/assert-from-live-cluster pattern.Closes#459
Note
Low Risk
Test-only change to
e2e-auto-upgrade.sh; no production Helm or chart behavior is modified.Overview
The fleet auto-upgrade e2e gate used one Helm release for both manual
--reuse-values(path 1) and fleet--reset-then-reuse-values(path 2), so path 1’s contamination of recorded values could make path 2’s prod pin assertion pass only when baseline and working-tree pins matched.Between path 1 and path 2, the script now runs
helm upgradeagainst the published baseline chart ($PREV) with--reset-valuesplus the three real install overrides (clientId,clientPassword,storageClass.provisioner), so new chart defaults are not pre-applied from the local tree. It fails ifimages.ingestor.prodDigeststill appears in user-supplied values (helm get valueswithout--all). Paths 2–4 then exercise clean-edge auto-upgrade behavior; comments replace the old ERA NOTE tripwire with #459 resolution notes.Reviewed by Cursor Bugbot for commit 5b6ee21. Bugbot is set up for automated code reviews on this repo. Configure here.