feat: make --three-way-merge work without patch permissions - #1063

Open
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission
Open

feat: make --three-way-merge work without patch permissions#1063
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission

Conversation

@oreonl

Copy link
Copy Markdown

What

--three-way-merge currently requires the patch permission on every diffed resource. It computes the merge patch locally but asks the API server to apply it as a dry-run:

helper.ServerDryRun=truetargetObj, err:=helper.Patch(info.Namespace, info.Name, patchType, patch, nil)

A read-only account gets cannot patch "x" with kind Deployment: ... is forbidden, so the feature is unavailable to exactly the credentials a "show me what this upgrade would do" tool tends to run under.

This applies the patch locally when the server round-trip is not available, using the same strategic-merge (or JSON merge patch, for custom resources) logic the API server would use. Only get is required.

Interface

A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE):

modebehaviour
auto (default)server dry-run, falling back to the local merge on Forbidden / MethodNotAllowed. Any other error still aborts, so a genuinely bad patch is not masked.
serverthe previous behaviour — a missing patch permission is a hard error.
clientnever sends the patch at all.

auto means a read-only account works out of the box; it only degrades when the accurate path is unavailable anyway, and says so on stderr.

manifest.Generate takes variadic options rather than a new parameter, so existing callers (helmfile vendors this package) keep compiling.

Matching what the server returns

A naive local merge produces a diff full of noise, because the API server does more than apply the patch. scheme.Scheme.Default() is a no-op in client-go — the SetDefaults_* functions live in k8s.io/kubernetes and are not importable — so the defaults cannot simply be recomputed. Two post-processing steps recover most of the gap:

1. Round-trip through the Go type, the way the API server does before it answers. That drops the empty values a manifest spells out but the type omits — initialDelaySeconds: 0, hostNetwork: false, sysctls: [], supplementalGroups: [] — which otherwise appear as additions the upgrade does not make. (Helm's kube.Client.Build returns unstructured objects, so these survive from the chart YAML verbatim.)

2. Copy a field back from the live object when the old and the new release manifest agree about it. Values disappear from the merged object two ways:

  • the patch replaces retainKeys structs (spec.strategy) and atomic lists (ports, volumeClaimTemplates) as a whole, dropping what the server had defaulted into them;
  • a manifest that renders an unset value as an explicit null — a bare replicas:, which is what a chart writes for a value it leaves unset — puts the key in both manifests, so it reaches the patch as a change rather than a deletion. That deletes a defaulted value even when the chart did not change at all.

The server re-defaults both cases immediately after applying the patch. Locally, a field is restored only when nothing in either manifest asked for it to go. Where the two manifests disagree the deletion is honoured, because that is a change the chart really makes — restoring there would hide real removals (a dropped nodeSelector entry looks identical at the JSON level, and the server does not default that back). Values already present in the merged object are never overwritten, so drift between the cluster and the chart is still reported and still corrected.

Known deviation from server mode

When the manifests genuinely disagree about a field the API server defaults, client mode reports it as removed where server mode shows it changing to the default — a chart that stops pinning replicas: 3 shows the field going away rather than 31. The change is reported either way; client mode just cannot name the value that replaces it. Mutating webhooks and validation are also not applied. All of this is documented in the new README section, along with the minimal RBAC role.

Tests

manifest/generate_test.go is new. Eight TestLocalMerge_* cases are built from real manifests (unstructured, as kube.Client.Build supplies them; the live side written as the API server returns it) and cover each symptom above, plus:

  • a chart that genuinely stops setting a field — still reported as removed;
  • a hand-scaled replicas and hand-edited image in the cluster — still reset to the chart values, i.e. drift detection intact.

I checked the tests are not vacuous by reverting each fix independently; each fails without the step it covers. cmd/upgrade_test.go adds flag/env validation cases.

go test ./..., go vet and gofmt pass, and the README flag tables are regenerated with make readme.

Testing done

Built with make install/helm and run against a real cluster with an account lacking patch, on a release of ~15 resources (LibreChat and its subcharts). Before the two post-processing steps the client-side output differed from --three-way-merge-mode=server on defaulted fields; the cases in the tests are taken from that output.

🤖 Generated with Claude Code

oreonland others added 2 commits August 31, 2026 11:23
--three-way-merge computed the merge patch locally but asked the API
server to apply it as a dry-run, which needs the `patch` verb on every
diffed resource. A read-only account got:
cannot patch "x" with kind Deployment: ... is forbidden
The patch is now applied locally when the server round-trip is not
available, using the same strategic-merge (or JSON merge patch, for
custom resources) logic the API server would use. Only `get` is
required.
A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE)
selects between:
auto server dry-run, falling back to the local merge on Forbidden
or MethodNotAllowed. Any other error still aborts, so a
genuinely bad patch is not masked. The default.
server the previous behaviour; a missing permission is a hard error.
client never sends the patch at all.
Two post-processing steps keep the local result close to what the API
server returns, since client-go does not ship the defaulting functions:
* The merged object is round-tripped through its Go type, the way the
API server does before it answers. That drops the empty values a
manifest spells out but the type omits - `initialDelaySeconds: 0`,
`hostNetwork: false`, `sysctls: []` - which would otherwise show up
as additions the upgrade does not make.
* A field is copied back from the live object when the old and the new
release manifest agree about it. The patch replaces `retainKeys`
structs and atomic lists as a whole, and a manifest that renders an
unset value as an explicit `null` (a bare `replicas:`) reaches the
patch as a change rather than a deletion; both wipe values the API
server had defaulted in and re-defaults immediately after. Fields
the two manifests disagree about are left deleted, because that is a
change the chart really makes.
manifest.Generate takes variadic options rather than a new parameter, so
existing callers keep compiling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A chart that renders an empty collection literally - `rules: []`, the
branch grafana's Role and ClusterRole take when no sidecar is enabled -
disagrees with the cluster about how to write "nothing". Kubernetes
stores objects as protobuf, which cannot tell an empty repeated field
from an absent one, so the API server answers with `rules: null`.
The patch then carries `{"rules":[]}` even though the chart did not
change, and the client-side merge reported:
rules (rbac.authorization.k8s.io/v1/Role/grafana/grafana)
± type change from <nil> to list
server mode shows nothing, because its answer goes back through the same
storage round-trip. The local merge now keeps whichever spelling the
cluster reports whenever the merged and the live value are both empty.
Emptying a collection that actually had entries is still a change and is
still reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

auto mode does not currently “fall back per run” (it keeps attempting forbidden PATCHes) and the docs/RBAC wording plus env-var handling need small corrections to match intended behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new three-way-merge “mode” capability so helm diff upgrade --three-way-merge can work with read-only Kubernetes credentials by falling back (or opting in) to a client-side patch application that approximates the API server’s results.

Changes:

  • Introduces --three-way-merge-mode (auto/server/client) and env HELM_DIFF_THREE_WAY_MERGE_MODE, wiring it through cmd/upgrade into manifest.Generate(...) via variadic options.
  • Implements client-side patch application for three-way-merge, including normalization (round-trip through Go types) and a restoration pass for server-populated/defaulted fields.
  • Adds focused unit tests for local merge behavior and CLI flag/env validation; updates README and Copilot instructions.
File summaries
FileDescription
README.mdDocuments --three-way-merge-mode, behavior tradeoffs, and RBAC guidance.
manifest/generate.goAdds merge-mode options and client-side patch application with post-processing to reduce diff noise.
manifest/generate_test.goNew tests covering strategic/merge patch local application and default-restoration behaviors.
cmd/upgrade.goAdds flag/env plumbing and validation; passes merge-mode into manifest generation.
cmd/upgrade_test.goAdds validation coverage for the new flag/env behavior.
.github/copilot-instructions.mdDocuments the new environment variable.
Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadmanifest/generate.go Outdated
Comment threadcmd/upgrade.go Outdated
Comment threadREADME.md Outdated
Comment threadmanifest/generate.go Outdated
auto mode now falls back for the whole run, as documented. It warned
once but kept sending a dry-run PATCH for every remaining resource, so a
release of N objects produced N denied requests - N audit log entries and
N round-trips - to reach the same conclusion each time. A refused dry-run
says something about the credentials rather than about the resource, so
the first refusal now moves the rest of the run to the local merge. The
decision moved into a clientSideFallback type so it can be tested without
a cluster.
HELM_DIFF_THREE_WAY_MERGE_MODE is read only when the run actually
performs a three-way merge (--three-way-merge, or --take-ownership which
implies it). A stale value in the environment could otherwise fail an
unrelated `helm diff upgrade` over a mode it was never going to use. The
flag stays validated unconditionally: passing it is deliberate, so a typo
is worth reporting either way.
The README no longer calls a wildcard role "minimal RBAC" - it grants
read access to every resource in every API group. It is now described as
the blunt option, next to a scoped example and a note that a kind missing
from the role fails the diff for that resource.
Also fixes the grammar in the two createPatch comments the review
flagged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new validation error messages format the allowed-mode slice with %q, which will render as %!q([]string=...) and should be corrected before merge.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

cmd/upgrade.go:199

  • The format string uses %q for manifest.ValidThreeWayMergeModes (a []string), which will render as %!q([]string=...) and produce an unhelpful validation error. Use %v (or join the slice) when printing the allowed values.

This issue also appears on line 226 of the same file.

cmd/upgrade.go:228

  • The env-var validation error formats manifest.ValidThreeWayMergeModes (a []string) with %q, which will print as %!q([]string=...). Use %v (or join the slice) so users see the allowed values clearly.
 if !slices.Contains(manifest.ValidThreeWayMergeModes, mode) {
return fmt.Errorf("env var %q must be one of %q, but got %q", "HELM_DIFF_THREE_WAY_MERGE_MODE", manifest.ValidThreeWayMergeModes, mode)
}
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@oreonl

Copy link
Copy Markdown
Author

Checked this one and I don't think it holds — %q on a []string isn't malformed. fmt applies a verb recursively to the elements of a compound operand (docs: "For compound objects, the elements are printed using these rules, recursively... array, slice: [elem0 elem1 ...]"). %!q(...) only appears when the verb is invalid for the type, e.g. %q on a struct.

Actual output from the current branch, for both lines flagged:

$ helm diff upgrade rel ./chart --three-way-merge-mode local
Error: flag "three-way-merge-mode" must be one of ["auto" "server" "client"], but got "local"
$ HELM_DIFF_THREE_WAY_MERGE_MODE=local helm diff upgrade rel ./chart --three-way-merge
Error: env var "HELM_DIFF_THREE_WAY_MERGE_MODE" must be one of ["auto" "server" "client"], but got "local"

%v would print [auto server client], dropping the quotes that delimit the values — and the adjacent --server-side check quotes each one:

Error: flag "server-side" must be "true", "false" or "auto", but got "yes"

So I've left these as they are.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core diff computation behavior with non-trivial Kubernetes patch/defaulting semantics, so it warrants final human review despite good test coverage.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment threadmanifest/generate.go Outdated
The marshal of originalObj reported "serializing current configuration",
while the live object marshalled a few lines below reported "serializing
live configuration". Two different objects, and the first was named after
the second, which is misleading when a failure has to be diagnosed. The
three messages now say original, target and live.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core three-way-merge behavior with substantial new merge/restoration logic whose correctness and compatibility impact are difficult to fully validate via automated review alone.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@yxxhero

Copy link
Copy Markdown
Collaborator

@oreonl PTAL

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Local restoration can hide real drift and CRD changes, while automatic fallback can mask admission failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadREADME.md Outdated
oreonland others added 2 commits September 3, 2026 13:55
Four defects from review, all confirmed by reproduction before the fix.
restoreMissing copied a live value back whenever the two release
manifests agreed about a key, including when they agreed on an actual
value. normalize drops an explicit `hostNetwork: false` through
omitempty, so a chart pinning false against a cluster that drifted to
true had the drift restored and the correction vanished from the diff -
the opposite of what three-way merge is for. A value is now restored only
where neither manifest carries one, an absent key and a `null` counting
alike, which is all the unset `replicas:` case ever needed.
The restoration pass no longer runs behind a JSON merge patch. That patch
carries only what the chart changed between the two manifests, so it
never prunes a field the cluster populated and there is nothing to put
back. Custom resources are also stored as JSON rather than protobuf,
which keeps `null`, `[]` and `{}` apart, so the empty-value equivalence
was undoing a chart deliberately changing one into another.
auto no longer reads every 403 as a missing patch permission. An
admission webhook or a quota controller can refuse a request the
credentials are entitled to make, and that refusal is exactly what the
upgrade would hit, so falling back locally hid the outcome the diff
exists to predict. A SelfSubjectAccessReview now settles whether patching
is permitted before the refusal is treated as a permission problem; where
the review cannot be made the previous behaviour stands. The fallback is
already once per run, so this costs one extra request.
The README documents a second deviation from server mode: changing an
entry inside an atomic list also drops the server defaults elsewhere in
that entry, because positions in a list the chart rewrote cannot be
matched up safely.
k8s.io/api/authorization/v1 is added to the depguard allowlist, which
golangci-lint requires for the new import.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fallback handling and field restoration have correctness gaps, and several usage and RBAC examples are inaccurate.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (5)

Previously missed (3) — in code that hasn't changed since the last review.

README.md:276

  • The release-storage requirement is understated here: the default revision is resolved by helm get manifest without --revision, which lists the Helm storage backend to find the newest release. A role with only get on Secrets/ConfigMaps therefore fails for the normal invocation; document list as required as well.

This issue also appears on line 297 of the same file.
README.md:404

  • This generated example does not enable three-way merge. Since the mode environment variable is intentionally ignored unless three-way merge is already active, the command performs a normal two-way diff and does not demonstrate client-side merging.
    cmd/upgrade.go:170
  • This example does not enable three-way merge. The command only reads this environment variable when HELM_DIFF_THREE_WAY_MERGE=true (or --three-way-merge/--take-ownership) is already active, so as written it performs the ordinary two-way diff rather than a local merge.

manifest/generate.go:233

  • MethodNotAllowed never falls back when the authorization review says patching is allowed: isPatchNotAllowed routes it here, then this check returns false and aborts. That contradicts the new auto contract, which explicitly falls back for MethodNotAllowed. Restrict the authorization disambiguation to Forbidden; a 405 indicates the server path is unavailable regardless of RBAC.
	if f.canPatch != nil {
if allowed, err := f.canPatch(info); err == nil && allowed {
return false
}

README.md:299

  • The least-privilege example cannot read the latest Helm release with the default revision. This code invokes helm get manifest without --revision (cmd/helm.go:139-158), so Helm must list the active Secret/ConfigMap storage backend to find the newest revision; granting only get causes the diff to fail before resource reads. Grant list to release storage while keeping workload kinds at get.
 - apiGroups: [""]
resources: ["configmaps", "secrets", "services", "serviceaccounts"]
verbs: ["get"]
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment threadmanifest/generate.go
A MethodNotAllowed no longer aborts. The authorization review added in
the previous commit ran for every refusal isPatchNotAllowed accepts, so a
405 against credentials that may patch was read as "not a permission
problem" and surfaced. A 405 says the API server takes no patch for the
resource whoever is asking, so being entitled to send one changes
nothing; only a 403 is ambiguous enough to be worth the review.
The least-privilege role could not read the release. Without an explicit
--revision helm lists the storage backend to find the newest one, so
`get` on Secrets and ConfigMaps is not enough - the release storage needs
`list` as well, while the kinds a chart renders still only need `get`.
The HELM_DIFF_THREE_WAY_MERGE_MODE examples did not enable the three-way
merge, so they demonstrated an ordinary two-way diff. Now that the
variable is deliberately ignored unless the merge is on, they have to
pass --three-way-merge to show anything. The README copy of the example
block is maintained by hand rather than by scripts/gen-readme.sh, which
only regenerates the flag tables, so it had drifted from the binary.
The README also documents that client mode can be quieter than the truth
in one place: the restoration copies back every live-only field a
wholesale replacement removed, and nothing local can tell a value the API
server defaulted from one somebody set by hand. A NetworkPolicy port with
a hand-added endPort keeps it, where the upgrade would drop it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oreonl

Copy link
Copy Markdown
Author

All five points from this review are addressed in d8d73ff. The four suppressed ones have no threads, so answering them here.

MethodNotAllowed aborting instead of falling back (manifest/generate.go:233) — correct, and a regression from the SelfSubjectAccessReview I added in ca059ad: the review ran for every refusal isPatchNotAllowed accepts, so a 405 against credentials that may patch was surfaced rather than falling back. Reproduced (patchDenied(405, canPatch=true) returned false) and fixed by restricting the disambiguation to Forbidden — a 405 says the API server takes no patch for that resource whoever is asking, so entitlement changes nothing. Covered by TestClientSideFallbackAlwaysFallsBackOnMethodNotAllowed.

Release storage needs list (README.md:276, :297, :299) — confirmed in the code: helmGetArgs omits --revision when the revision is 0 (cmd/helm.go:141-145), and helm's Secrets driver Query does a List to find the newest revision, so get alone fails before any resource is read. The least-privilege example now grants get, list on configmaps/secrets for the release storage and keeps the rendered kinds at get, with the reason spelled out in the surrounding prose.

The env var examples did not enable the three-way merge (README.md:404, cmd/upgrade.go:170) — right, and self-inflicted: making HELM_DIFF_THREE_WAY_MERGE_MODE deliberately ignored unless the merge is on invalidated my own examples. Both now pass --three-way-merge. Worth flagging for future edits that scripts/gen-readme.sh regenerates only the flag tables, so the README's copy of the example block is hand-maintained and had drifted from the binary; I diffed the two to confirm they match again.

The remaining inline comment, about live-only drift inside a replaced atomic list, is answered on its own thread. I have left that one open, since it is a design trade-off rather than something I have fixed.

For what it is worth while the checks are still pending: golangci-lint v2.12.2 against Go 1.26.4 — the pairing lint.yaml pins via go-version-file — reports 0 issues, and go test ./... passes.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Local normalization can discard newer Kubernetes fields, while explicit-null restoration can hide real drift corrections.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

manifest/generate.go:365

  • Decoding into the compiled Go type silently discards fields unknown to this client-go version. If helm-diff runs against a newer cluster and a chart adds a newly supported field to an otherwise known built-in kind, the local merge contains the field here, this round-trip removes it, and the restoration pass cannot recover it because it is absent from the live object. Client mode then omits a real upgrade change; preserve unknown fields from the merged object during normalization or avoid typed normalization when the typed schema does not cover the object.

manifest/generate.go:474

  • This treats an explicit null exactly like a missing key, but null can be the chart-owned desired value for fields that are not defaulted. For example, if both manifests render runtimeClassName: null while the live Pod template drifted to runtimeClassName: gvisor, the three-way patch clears the field and normalize omits it; this condition then restores gvisor, hiding a correction that the real upgrade makes. Use presence-aware lookups and restrict null restoration to fields known to be server-defaulted rather than restoring every explicit-null field.
 if originalMap[key] == nil && modifiedMap[key] == nil {
mergedMap[key] = liveValue
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@oreonl@yxxhero
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

feat: make --three-way-merge work without patch permissions - #1063

Open
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission
Open

feat: make --three-way-merge work without patch permissions#1063
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission

Conversation

@oreonl

Copy link
Copy Markdown

What

--three-way-merge currently requires the patch permission on every diffed resource. It computes the merge patch locally but asks the API server to apply it as a dry-run:

helper.ServerDryRun=truetargetObj, err:=helper.Patch(info.Namespace, info.Name, patchType, patch, nil)

A read-only account gets cannot patch "x" with kind Deployment: ... is forbidden, so the feature is unavailable to exactly the credentials a "show me what this upgrade would do" tool tends to run under.

This applies the patch locally when the server round-trip is not available, using the same strategic-merge (or JSON merge patch, for custom resources) logic the API server would use. Only get is required.

Interface

A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE):

modebehaviour
auto (default)server dry-run, falling back to the local merge on Forbidden / MethodNotAllowed. Any other error still aborts, so a genuinely bad patch is not masked.
serverthe previous behaviour — a missing patch permission is a hard error.
clientnever sends the patch at all.

auto means a read-only account works out of the box; it only degrades when the accurate path is unavailable anyway, and says so on stderr.

manifest.Generate takes variadic options rather than a new parameter, so existing callers (helmfile vendors this package) keep compiling.

Matching what the server returns

A naive local merge produces a diff full of noise, because the API server does more than apply the patch. scheme.Scheme.Default() is a no-op in client-go — the SetDefaults_* functions live in k8s.io/kubernetes and are not importable — so the defaults cannot simply be recomputed. Two post-processing steps recover most of the gap:

1. Round-trip through the Go type, the way the API server does before it answers. That drops the empty values a manifest spells out but the type omits — initialDelaySeconds: 0, hostNetwork: false, sysctls: [], supplementalGroups: [] — which otherwise appear as additions the upgrade does not make. (Helm's kube.Client.Build returns unstructured objects, so these survive from the chart YAML verbatim.)

2. Copy a field back from the live object when the old and the new release manifest agree about it. Values disappear from the merged object two ways:

  • the patch replaces retainKeys structs (spec.strategy) and atomic lists (ports, volumeClaimTemplates) as a whole, dropping what the server had defaulted into them;
  • a manifest that renders an unset value as an explicit null — a bare replicas:, which is what a chart writes for a value it leaves unset — puts the key in both manifests, so it reaches the patch as a change rather than a deletion. That deletes a defaulted value even when the chart did not change at all.

The server re-defaults both cases immediately after applying the patch. Locally, a field is restored only when nothing in either manifest asked for it to go. Where the two manifests disagree the deletion is honoured, because that is a change the chart really makes — restoring there would hide real removals (a dropped nodeSelector entry looks identical at the JSON level, and the server does not default that back). Values already present in the merged object are never overwritten, so drift between the cluster and the chart is still reported and still corrected.

Known deviation from server mode

When the manifests genuinely disagree about a field the API server defaults, client mode reports it as removed where server mode shows it changing to the default — a chart that stops pinning replicas: 3 shows the field going away rather than 31. The change is reported either way; client mode just cannot name the value that replaces it. Mutating webhooks and validation are also not applied. All of this is documented in the new README section, along with the minimal RBAC role.

Tests

manifest/generate_test.go is new. Eight TestLocalMerge_* cases are built from real manifests (unstructured, as kube.Client.Build supplies them; the live side written as the API server returns it) and cover each symptom above, plus:

  • a chart that genuinely stops setting a field — still reported as removed;
  • a hand-scaled replicas and hand-edited image in the cluster — still reset to the chart values, i.e. drift detection intact.

I checked the tests are not vacuous by reverting each fix independently; each fails without the step it covers. cmd/upgrade_test.go adds flag/env validation cases.

go test ./..., go vet and gofmt pass, and the README flag tables are regenerated with make readme.

Testing done

Built with make install/helm and run against a real cluster with an account lacking patch, on a release of ~15 resources (LibreChat and its subcharts). Before the two post-processing steps the client-side output differed from --three-way-merge-mode=server on defaulted fields; the cases in the tests are taken from that output.

🤖 Generated with Claude Code

oreonland others added 2 commits August 31, 2026 11:23
--three-way-merge computed the merge patch locally but asked the API
server to apply it as a dry-run, which needs the `patch` verb on every
diffed resource. A read-only account got:
cannot patch "x" with kind Deployment: ... is forbidden
The patch is now applied locally when the server round-trip is not
available, using the same strategic-merge (or JSON merge patch, for
custom resources) logic the API server would use. Only `get` is
required.
A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE)
selects between:
auto server dry-run, falling back to the local merge on Forbidden
or MethodNotAllowed. Any other error still aborts, so a
genuinely bad patch is not masked. The default.
server the previous behaviour; a missing permission is a hard error.
client never sends the patch at all.
Two post-processing steps keep the local result close to what the API
server returns, since client-go does not ship the defaulting functions:
* The merged object is round-tripped through its Go type, the way the
API server does before it answers. That drops the empty values a
manifest spells out but the type omits - `initialDelaySeconds: 0`,
`hostNetwork: false`, `sysctls: []` - which would otherwise show up
as additions the upgrade does not make.
* A field is copied back from the live object when the old and the new
release manifest agree about it. The patch replaces `retainKeys`
structs and atomic lists as a whole, and a manifest that renders an
unset value as an explicit `null` (a bare `replicas:`) reaches the
patch as a change rather than a deletion; both wipe values the API
server had defaulted in and re-defaults immediately after. Fields
the two manifests disagree about are left deleted, because that is a
change the chart really makes.
manifest.Generate takes variadic options rather than a new parameter, so
existing callers keep compiling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A chart that renders an empty collection literally - `rules: []`, the
branch grafana's Role and ClusterRole take when no sidecar is enabled -
disagrees with the cluster about how to write "nothing". Kubernetes
stores objects as protobuf, which cannot tell an empty repeated field
from an absent one, so the API server answers with `rules: null`.
The patch then carries `{"rules":[]}` even though the chart did not
change, and the client-side merge reported:
rules (rbac.authorization.k8s.io/v1/Role/grafana/grafana)
± type change from <nil> to list
server mode shows nothing, because its answer goes back through the same
storage round-trip. The local merge now keeps whichever spelling the
cluster reports whenever the merged and the live value are both empty.
Emptying a collection that actually had entries is still a change and is
still reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

auto mode does not currently “fall back per run” (it keeps attempting forbidden PATCHes) and the docs/RBAC wording plus env-var handling need small corrections to match intended behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new three-way-merge “mode” capability so helm diff upgrade --three-way-merge can work with read-only Kubernetes credentials by falling back (or opting in) to a client-side patch application that approximates the API server’s results.

Changes:

  • Introduces --three-way-merge-mode (auto/server/client) and env HELM_DIFF_THREE_WAY_MERGE_MODE, wiring it through cmd/upgrade into manifest.Generate(...) via variadic options.
  • Implements client-side patch application for three-way-merge, including normalization (round-trip through Go types) and a restoration pass for server-populated/defaulted fields.
  • Adds focused unit tests for local merge behavior and CLI flag/env validation; updates README and Copilot instructions.
File summaries
FileDescription
README.mdDocuments --three-way-merge-mode, behavior tradeoffs, and RBAC guidance.
manifest/generate.goAdds merge-mode options and client-side patch application with post-processing to reduce diff noise.
manifest/generate_test.goNew tests covering strategic/merge patch local application and default-restoration behaviors.
cmd/upgrade.goAdds flag/env plumbing and validation; passes merge-mode into manifest generation.
cmd/upgrade_test.goAdds validation coverage for the new flag/env behavior.
.github/copilot-instructions.mdDocuments the new environment variable.
Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadmanifest/generate.go Outdated
Comment threadcmd/upgrade.go Outdated
Comment threadREADME.md Outdated
Comment threadmanifest/generate.go Outdated
auto mode now falls back for the whole run, as documented. It warned
once but kept sending a dry-run PATCH for every remaining resource, so a
release of N objects produced N denied requests - N audit log entries and
N round-trips - to reach the same conclusion each time. A refused dry-run
says something about the credentials rather than about the resource, so
the first refusal now moves the rest of the run to the local merge. The
decision moved into a clientSideFallback type so it can be tested without
a cluster.
HELM_DIFF_THREE_WAY_MERGE_MODE is read only when the run actually
performs a three-way merge (--three-way-merge, or --take-ownership which
implies it). A stale value in the environment could otherwise fail an
unrelated `helm diff upgrade` over a mode it was never going to use. The
flag stays validated unconditionally: passing it is deliberate, so a typo
is worth reporting either way.
The README no longer calls a wildcard role "minimal RBAC" - it grants
read access to every resource in every API group. It is now described as
the blunt option, next to a scoped example and a note that a kind missing
from the role fails the diff for that resource.
Also fixes the grammar in the two createPatch comments the review
flagged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new validation error messages format the allowed-mode slice with %q, which will render as %!q([]string=...) and should be corrected before merge.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

cmd/upgrade.go:199

  • The format string uses %q for manifest.ValidThreeWayMergeModes (a []string), which will render as %!q([]string=...) and produce an unhelpful validation error. Use %v (or join the slice) when printing the allowed values.

This issue also appears on line 226 of the same file.

cmd/upgrade.go:228

  • The env-var validation error formats manifest.ValidThreeWayMergeModes (a []string) with %q, which will print as %!q([]string=...). Use %v (or join the slice) so users see the allowed values clearly.
 if !slices.Contains(manifest.ValidThreeWayMergeModes, mode) {
return fmt.Errorf("env var %q must be one of %q, but got %q", "HELM_DIFF_THREE_WAY_MERGE_MODE", manifest.ValidThreeWayMergeModes, mode)
}
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@oreonl

Copy link
Copy Markdown
Author

Checked this one and I don't think it holds — %q on a []string isn't malformed. fmt applies a verb recursively to the elements of a compound operand (docs: "For compound objects, the elements are printed using these rules, recursively... array, slice: [elem0 elem1 ...]"). %!q(...) only appears when the verb is invalid for the type, e.g. %q on a struct.

Actual output from the current branch, for both lines flagged:

$ helm diff upgrade rel ./chart --three-way-merge-mode local
Error: flag "three-way-merge-mode" must be one of ["auto" "server" "client"], but got "local"
$ HELM_DIFF_THREE_WAY_MERGE_MODE=local helm diff upgrade rel ./chart --three-way-merge
Error: env var "HELM_DIFF_THREE_WAY_MERGE_MODE" must be one of ["auto" "server" "client"], but got "local"

%v would print [auto server client], dropping the quotes that delimit the values — and the adjacent --server-side check quotes each one:

Error: flag "server-side" must be "true", "false" or "auto", but got "yes"

So I've left these as they are.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core diff computation behavior with non-trivial Kubernetes patch/defaulting semantics, so it warrants final human review despite good test coverage.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment threadmanifest/generate.go Outdated
The marshal of originalObj reported "serializing current configuration",
while the live object marshalled a few lines below reported "serializing
live configuration". Two different objects, and the first was named after
the second, which is misleading when a failure has to be diagnosed. The
three messages now say original, target and live.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core three-way-merge behavior with substantial new merge/restoration logic whose correctness and compatibility impact are difficult to fully validate via automated review alone.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@yxxhero

Copy link
Copy Markdown
Collaborator

@oreonl PTAL

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Local restoration can hide real drift and CRD changes, while automatic fallback can mask admission failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadREADME.md Outdated
oreonland others added 2 commits September 3, 2026 13:55
Four defects from review, all confirmed by reproduction before the fix.
restoreMissing copied a live value back whenever the two release
manifests agreed about a key, including when they agreed on an actual
value. normalize drops an explicit `hostNetwork: false` through
omitempty, so a chart pinning false against a cluster that drifted to
true had the drift restored and the correction vanished from the diff -
the opposite of what three-way merge is for. A value is now restored only
where neither manifest carries one, an absent key and a `null` counting
alike, which is all the unset `replicas:` case ever needed.
The restoration pass no longer runs behind a JSON merge patch. That patch
carries only what the chart changed between the two manifests, so it
never prunes a field the cluster populated and there is nothing to put
back. Custom resources are also stored as JSON rather than protobuf,
which keeps `null`, `[]` and `{}` apart, so the empty-value equivalence
was undoing a chart deliberately changing one into another.
auto no longer reads every 403 as a missing patch permission. An
admission webhook or a quota controller can refuse a request the
credentials are entitled to make, and that refusal is exactly what the
upgrade would hit, so falling back locally hid the outcome the diff
exists to predict. A SelfSubjectAccessReview now settles whether patching
is permitted before the refusal is treated as a permission problem; where
the review cannot be made the previous behaviour stands. The fallback is
already once per run, so this costs one extra request.
The README documents a second deviation from server mode: changing an
entry inside an atomic list also drops the server defaults elsewhere in
that entry, because positions in a list the chart rewrote cannot be
matched up safely.
k8s.io/api/authorization/v1 is added to the depguard allowlist, which
golangci-lint requires for the new import.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fallback handling and field restoration have correctness gaps, and several usage and RBAC examples are inaccurate.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (5)

Previously missed (3) — in code that hasn't changed since the last review.

README.md:276

  • The release-storage requirement is understated here: the default revision is resolved by helm get manifest without --revision, which lists the Helm storage backend to find the newest release. A role with only get on Secrets/ConfigMaps therefore fails for the normal invocation; document list as required as well.

This issue also appears on line 297 of the same file.
README.md:404

  • This generated example does not enable three-way merge. Since the mode environment variable is intentionally ignored unless three-way merge is already active, the command performs a normal two-way diff and does not demonstrate client-side merging.
    cmd/upgrade.go:170
  • This example does not enable three-way merge. The command only reads this environment variable when HELM_DIFF_THREE_WAY_MERGE=true (or --three-way-merge/--take-ownership) is already active, so as written it performs the ordinary two-way diff rather than a local merge.

manifest/generate.go:233

  • MethodNotAllowed never falls back when the authorization review says patching is allowed: isPatchNotAllowed routes it here, then this check returns false and aborts. That contradicts the new auto contract, which explicitly falls back for MethodNotAllowed. Restrict the authorization disambiguation to Forbidden; a 405 indicates the server path is unavailable regardless of RBAC.
	if f.canPatch != nil {
if allowed, err := f.canPatch(info); err == nil && allowed {
return false
}

README.md:299

  • The least-privilege example cannot read the latest Helm release with the default revision. This code invokes helm get manifest without --revision (cmd/helm.go:139-158), so Helm must list the active Secret/ConfigMap storage backend to find the newest revision; granting only get causes the diff to fail before resource reads. Grant list to release storage while keeping workload kinds at get.
 - apiGroups: [""]
resources: ["configmaps", "secrets", "services", "serviceaccounts"]
verbs: ["get"]
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment threadmanifest/generate.go
A MethodNotAllowed no longer aborts. The authorization review added in
the previous commit ran for every refusal isPatchNotAllowed accepts, so a
405 against credentials that may patch was read as "not a permission
problem" and surfaced. A 405 says the API server takes no patch for the
resource whoever is asking, so being entitled to send one changes
nothing; only a 403 is ambiguous enough to be worth the review.
The least-privilege role could not read the release. Without an explicit
--revision helm lists the storage backend to find the newest one, so
`get` on Secrets and ConfigMaps is not enough - the release storage needs
`list` as well, while the kinds a chart renders still only need `get`.
The HELM_DIFF_THREE_WAY_MERGE_MODE examples did not enable the three-way
merge, so they demonstrated an ordinary two-way diff. Now that the
variable is deliberately ignored unless the merge is on, they have to
pass --three-way-merge to show anything. The README copy of the example
block is maintained by hand rather than by scripts/gen-readme.sh, which
only regenerates the flag tables, so it had drifted from the binary.
The README also documents that client mode can be quieter than the truth
in one place: the restoration copies back every live-only field a
wholesale replacement removed, and nothing local can tell a value the API
server defaulted from one somebody set by hand. A NetworkPolicy port with
a hand-added endPort keeps it, where the upgrade would drop it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oreonl

Copy link
Copy Markdown
Author

All five points from this review are addressed in d8d73ff. The four suppressed ones have no threads, so answering them here.

MethodNotAllowed aborting instead of falling back (manifest/generate.go:233) — correct, and a regression from the SelfSubjectAccessReview I added in ca059ad: the review ran for every refusal isPatchNotAllowed accepts, so a 405 against credentials that may patch was surfaced rather than falling back. Reproduced (patchDenied(405, canPatch=true) returned false) and fixed by restricting the disambiguation to Forbidden — a 405 says the API server takes no patch for that resource whoever is asking, so entitlement changes nothing. Covered by TestClientSideFallbackAlwaysFallsBackOnMethodNotAllowed.

Release storage needs list (README.md:276, :297, :299) — confirmed in the code: helmGetArgs omits --revision when the revision is 0 (cmd/helm.go:141-145), and helm's Secrets driver Query does a List to find the newest revision, so get alone fails before any resource is read. The least-privilege example now grants get, list on configmaps/secrets for the release storage and keeps the rendered kinds at get, with the reason spelled out in the surrounding prose.

The env var examples did not enable the three-way merge (README.md:404, cmd/upgrade.go:170) — right, and self-inflicted: making HELM_DIFF_THREE_WAY_MERGE_MODE deliberately ignored unless the merge is on invalidated my own examples. Both now pass --three-way-merge. Worth flagging for future edits that scripts/gen-readme.sh regenerates only the flag tables, so the README's copy of the example block is hand-maintained and had drifted from the binary; I diffed the two to confirm they match again.

The remaining inline comment, about live-only drift inside a replaced atomic list, is answered on its own thread. I have left that one open, since it is a design trade-off rather than something I have fixed.

For what it is worth while the checks are still pending: golangci-lint v2.12.2 against Go 1.26.4 — the pairing lint.yaml pins via go-version-file — reports 0 issues, and go test ./... passes.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Local normalization can discard newer Kubernetes fields, while explicit-null restoration can hide real drift corrections.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

manifest/generate.go:365

  • Decoding into the compiled Go type silently discards fields unknown to this client-go version. If helm-diff runs against a newer cluster and a chart adds a newly supported field to an otherwise known built-in kind, the local merge contains the field here, this round-trip removes it, and the restoration pass cannot recover it because it is absent from the live object. Client mode then omits a real upgrade change; preserve unknown fields from the merged object during normalization or avoid typed normalization when the typed schema does not cover the object.

manifest/generate.go:474

  • This treats an explicit null exactly like a missing key, but null can be the chart-owned desired value for fields that are not defaulted. For example, if both manifests render runtimeClassName: null while the live Pod template drifted to runtimeClassName: gvisor, the three-way patch clears the field and normalize omits it; this condition then restores gvisor, hiding a correction that the real upgrade makes. Use presence-aware lookups and restrict null restoration to fields known to be server-defaulted rather than restoring every explicit-null field.
 if originalMap[key] == nil && modifiedMap[key] == nil {
mergedMap[key] = liveValue
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@oreonl@yxxhero
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat: make --three-way-merge work without patch permissions - #1063

Open
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission
Open

feat: make --three-way-merge work without patch permissions#1063
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission

Conversation

@oreonl

Copy link
Copy Markdown

What

--three-way-merge currently requires the patch permission on every diffed resource. It computes the merge patch locally but asks the API server to apply it as a dry-run:

helper.ServerDryRun=truetargetObj, err:=helper.Patch(info.Namespace, info.Name, patchType, patch, nil)

A read-only account gets cannot patch "x" with kind Deployment: ... is forbidden, so the feature is unavailable to exactly the credentials a "show me what this upgrade would do" tool tends to run under.

This applies the patch locally when the server round-trip is not available, using the same strategic-merge (or JSON merge patch, for custom resources) logic the API server would use. Only get is required.

Interface

A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE):

modebehaviour
auto (default)server dry-run, falling back to the local merge on Forbidden / MethodNotAllowed. Any other error still aborts, so a genuinely bad patch is not masked.
serverthe previous behaviour — a missing patch permission is a hard error.
clientnever sends the patch at all.

auto means a read-only account works out of the box; it only degrades when the accurate path is unavailable anyway, and says so on stderr.

manifest.Generate takes variadic options rather than a new parameter, so existing callers (helmfile vendors this package) keep compiling.

Matching what the server returns

A naive local merge produces a diff full of noise, because the API server does more than apply the patch. scheme.Scheme.Default() is a no-op in client-go — the SetDefaults_* functions live in k8s.io/kubernetes and are not importable — so the defaults cannot simply be recomputed. Two post-processing steps recover most of the gap:

1. Round-trip through the Go type, the way the API server does before it answers. That drops the empty values a manifest spells out but the type omits — initialDelaySeconds: 0, hostNetwork: false, sysctls: [], supplementalGroups: [] — which otherwise appear as additions the upgrade does not make. (Helm's kube.Client.Build returns unstructured objects, so these survive from the chart YAML verbatim.)

2. Copy a field back from the live object when the old and the new release manifest agree about it. Values disappear from the merged object two ways:

  • the patch replaces retainKeys structs (spec.strategy) and atomic lists (ports, volumeClaimTemplates) as a whole, dropping what the server had defaulted into them;
  • a manifest that renders an unset value as an explicit null — a bare replicas:, which is what a chart writes for a value it leaves unset — puts the key in both manifests, so it reaches the patch as a change rather than a deletion. That deletes a defaulted value even when the chart did not change at all.

The server re-defaults both cases immediately after applying the patch. Locally, a field is restored only when nothing in either manifest asked for it to go. Where the two manifests disagree the deletion is honoured, because that is a change the chart really makes — restoring there would hide real removals (a dropped nodeSelector entry looks identical at the JSON level, and the server does not default that back). Values already present in the merged object are never overwritten, so drift between the cluster and the chart is still reported and still corrected.

Known deviation from server mode

When the manifests genuinely disagree about a field the API server defaults, client mode reports it as removed where server mode shows it changing to the default — a chart that stops pinning replicas: 3 shows the field going away rather than 31. The change is reported either way; client mode just cannot name the value that replaces it. Mutating webhooks and validation are also not applied. All of this is documented in the new README section, along with the minimal RBAC role.

Tests

manifest/generate_test.go is new. Eight TestLocalMerge_* cases are built from real manifests (unstructured, as kube.Client.Build supplies them; the live side written as the API server returns it) and cover each symptom above, plus:

  • a chart that genuinely stops setting a field — still reported as removed;
  • a hand-scaled replicas and hand-edited image in the cluster — still reset to the chart values, i.e. drift detection intact.

I checked the tests are not vacuous by reverting each fix independently; each fails without the step it covers. cmd/upgrade_test.go adds flag/env validation cases.

go test ./..., go vet and gofmt pass, and the README flag tables are regenerated with make readme.

Testing done

Built with make install/helm and run against a real cluster with an account lacking patch, on a release of ~15 resources (LibreChat and its subcharts). Before the two post-processing steps the client-side output differed from --three-way-merge-mode=server on defaulted fields; the cases in the tests are taken from that output.

🤖 Generated with Claude Code

oreonland others added 2 commits August 31, 2026 11:23
--three-way-merge computed the merge patch locally but asked the API
server to apply it as a dry-run, which needs the `patch` verb on every
diffed resource. A read-only account got:
cannot patch "x" with kind Deployment: ... is forbidden
The patch is now applied locally when the server round-trip is not
available, using the same strategic-merge (or JSON merge patch, for
custom resources) logic the API server would use. Only `get` is
required.
A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE)
selects between:
auto server dry-run, falling back to the local merge on Forbidden
or MethodNotAllowed. Any other error still aborts, so a
genuinely bad patch is not masked. The default.
server the previous behaviour; a missing permission is a hard error.
client never sends the patch at all.
Two post-processing steps keep the local result close to what the API
server returns, since client-go does not ship the defaulting functions:
* The merged object is round-tripped through its Go type, the way the
API server does before it answers. That drops the empty values a
manifest spells out but the type omits - `initialDelaySeconds: 0`,
`hostNetwork: false`, `sysctls: []` - which would otherwise show up
as additions the upgrade does not make.
* A field is copied back from the live object when the old and the new
release manifest agree about it. The patch replaces `retainKeys`
structs and atomic lists as a whole, and a manifest that renders an
unset value as an explicit `null` (a bare `replicas:`) reaches the
patch as a change rather than a deletion; both wipe values the API
server had defaulted in and re-defaults immediately after. Fields
the two manifests disagree about are left deleted, because that is a
change the chart really makes.
manifest.Generate takes variadic options rather than a new parameter, so
existing callers keep compiling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A chart that renders an empty collection literally - `rules: []`, the
branch grafana's Role and ClusterRole take when no sidecar is enabled -
disagrees with the cluster about how to write "nothing". Kubernetes
stores objects as protobuf, which cannot tell an empty repeated field
from an absent one, so the API server answers with `rules: null`.
The patch then carries `{"rules":[]}` even though the chart did not
change, and the client-side merge reported:
rules (rbac.authorization.k8s.io/v1/Role/grafana/grafana)
± type change from <nil> to list
server mode shows nothing, because its answer goes back through the same
storage round-trip. The local merge now keeps whichever spelling the
cluster reports whenever the merged and the live value are both empty.
Emptying a collection that actually had entries is still a change and is
still reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

auto mode does not currently “fall back per run” (it keeps attempting forbidden PATCHes) and the docs/RBAC wording plus env-var handling need small corrections to match intended behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new three-way-merge “mode” capability so helm diff upgrade --three-way-merge can work with read-only Kubernetes credentials by falling back (or opting in) to a client-side patch application that approximates the API server’s results.

Changes:

  • Introduces --three-way-merge-mode (auto/server/client) and env HELM_DIFF_THREE_WAY_MERGE_MODE, wiring it through cmd/upgrade into manifest.Generate(...) via variadic options.
  • Implements client-side patch application for three-way-merge, including normalization (round-trip through Go types) and a restoration pass for server-populated/defaulted fields.
  • Adds focused unit tests for local merge behavior and CLI flag/env validation; updates README and Copilot instructions.
File summaries
FileDescription
README.mdDocuments --three-way-merge-mode, behavior tradeoffs, and RBAC guidance.
manifest/generate.goAdds merge-mode options and client-side patch application with post-processing to reduce diff noise.
manifest/generate_test.goNew tests covering strategic/merge patch local application and default-restoration behaviors.
cmd/upgrade.goAdds flag/env plumbing and validation; passes merge-mode into manifest generation.
cmd/upgrade_test.goAdds validation coverage for the new flag/env behavior.
.github/copilot-instructions.mdDocuments the new environment variable.
Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadmanifest/generate.go Outdated
Comment threadcmd/upgrade.go Outdated
Comment threadREADME.md Outdated
Comment threadmanifest/generate.go Outdated
auto mode now falls back for the whole run, as documented. It warned
once but kept sending a dry-run PATCH for every remaining resource, so a
release of N objects produced N denied requests - N audit log entries and
N round-trips - to reach the same conclusion each time. A refused dry-run
says something about the credentials rather than about the resource, so
the first refusal now moves the rest of the run to the local merge. The
decision moved into a clientSideFallback type so it can be tested without
a cluster.
HELM_DIFF_THREE_WAY_MERGE_MODE is read only when the run actually
performs a three-way merge (--three-way-merge, or --take-ownership which
implies it). A stale value in the environment could otherwise fail an
unrelated `helm diff upgrade` over a mode it was never going to use. The
flag stays validated unconditionally: passing it is deliberate, so a typo
is worth reporting either way.
The README no longer calls a wildcard role "minimal RBAC" - it grants
read access to every resource in every API group. It is now described as
the blunt option, next to a scoped example and a note that a kind missing
from the role fails the diff for that resource.
Also fixes the grammar in the two createPatch comments the review
flagged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new validation error messages format the allowed-mode slice with %q, which will render as %!q([]string=...) and should be corrected before merge.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

cmd/upgrade.go:199

  • The format string uses %q for manifest.ValidThreeWayMergeModes (a []string), which will render as %!q([]string=...) and produce an unhelpful validation error. Use %v (or join the slice) when printing the allowed values.

This issue also appears on line 226 of the same file.

cmd/upgrade.go:228

  • The env-var validation error formats manifest.ValidThreeWayMergeModes (a []string) with %q, which will print as %!q([]string=...). Use %v (or join the slice) so users see the allowed values clearly.
 if !slices.Contains(manifest.ValidThreeWayMergeModes, mode) {
return fmt.Errorf("env var %q must be one of %q, but got %q", "HELM_DIFF_THREE_WAY_MERGE_MODE", manifest.ValidThreeWayMergeModes, mode)
}
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@oreonl

Copy link
Copy Markdown
Author

Checked this one and I don't think it holds — %q on a []string isn't malformed. fmt applies a verb recursively to the elements of a compound operand (docs: "For compound objects, the elements are printed using these rules, recursively... array, slice: [elem0 elem1 ...]"). %!q(...) only appears when the verb is invalid for the type, e.g. %q on a struct.

Actual output from the current branch, for both lines flagged:

$ helm diff upgrade rel ./chart --three-way-merge-mode local
Error: flag "three-way-merge-mode" must be one of ["auto" "server" "client"], but got "local"
$ HELM_DIFF_THREE_WAY_MERGE_MODE=local helm diff upgrade rel ./chart --three-way-merge
Error: env var "HELM_DIFF_THREE_WAY_MERGE_MODE" must be one of ["auto" "server" "client"], but got "local"

%v would print [auto server client], dropping the quotes that delimit the values — and the adjacent --server-side check quotes each one:

Error: flag "server-side" must be "true", "false" or "auto", but got "yes"

So I've left these as they are.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core diff computation behavior with non-trivial Kubernetes patch/defaulting semantics, so it warrants final human review despite good test coverage.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment threadmanifest/generate.go Outdated
The marshal of originalObj reported "serializing current configuration",
while the live object marshalled a few lines below reported "serializing
live configuration". Two different objects, and the first was named after
the second, which is misleading when a failure has to be diagnosed. The
three messages now say original, target and live.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core three-way-merge behavior with substantial new merge/restoration logic whose correctness and compatibility impact are difficult to fully validate via automated review alone.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@yxxhero

Copy link
Copy Markdown
Collaborator

@oreonl PTAL

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Local restoration can hide real drift and CRD changes, while automatic fallback can mask admission failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadREADME.md Outdated
oreonland others added 2 commits September 3, 2026 13:55
Four defects from review, all confirmed by reproduction before the fix.
restoreMissing copied a live value back whenever the two release
manifests agreed about a key, including when they agreed on an actual
value. normalize drops an explicit `hostNetwork: false` through
omitempty, so a chart pinning false against a cluster that drifted to
true had the drift restored and the correction vanished from the diff -
the opposite of what three-way merge is for. A value is now restored only
where neither manifest carries one, an absent key and a `null` counting
alike, which is all the unset `replicas:` case ever needed.
The restoration pass no longer runs behind a JSON merge patch. That patch
carries only what the chart changed between the two manifests, so it
never prunes a field the cluster populated and there is nothing to put
back. Custom resources are also stored as JSON rather than protobuf,
which keeps `null`, `[]` and `{}` apart, so the empty-value equivalence
was undoing a chart deliberately changing one into another.
auto no longer reads every 403 as a missing patch permission. An
admission webhook or a quota controller can refuse a request the
credentials are entitled to make, and that refusal is exactly what the
upgrade would hit, so falling back locally hid the outcome the diff
exists to predict. A SelfSubjectAccessReview now settles whether patching
is permitted before the refusal is treated as a permission problem; where
the review cannot be made the previous behaviour stands. The fallback is
already once per run, so this costs one extra request.
The README documents a second deviation from server mode: changing an
entry inside an atomic list also drops the server defaults elsewhere in
that entry, because positions in a list the chart rewrote cannot be
matched up safely.
k8s.io/api/authorization/v1 is added to the depguard allowlist, which
golangci-lint requires for the new import.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fallback handling and field restoration have correctness gaps, and several usage and RBAC examples are inaccurate.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (5)

Previously missed (3) — in code that hasn't changed since the last review.

README.md:276

  • The release-storage requirement is understated here: the default revision is resolved by helm get manifest without --revision, which lists the Helm storage backend to find the newest release. A role with only get on Secrets/ConfigMaps therefore fails for the normal invocation; document list as required as well.

This issue also appears on line 297 of the same file.
README.md:404

  • This generated example does not enable three-way merge. Since the mode environment variable is intentionally ignored unless three-way merge is already active, the command performs a normal two-way diff and does not demonstrate client-side merging.
    cmd/upgrade.go:170
  • This example does not enable three-way merge. The command only reads this environment variable when HELM_DIFF_THREE_WAY_MERGE=true (or --three-way-merge/--take-ownership) is already active, so as written it performs the ordinary two-way diff rather than a local merge.

manifest/generate.go:233

  • MethodNotAllowed never falls back when the authorization review says patching is allowed: isPatchNotAllowed routes it here, then this check returns false and aborts. That contradicts the new auto contract, which explicitly falls back for MethodNotAllowed. Restrict the authorization disambiguation to Forbidden; a 405 indicates the server path is unavailable regardless of RBAC.
	if f.canPatch != nil {
if allowed, err := f.canPatch(info); err == nil && allowed {
return false
}

README.md:299

  • The least-privilege example cannot read the latest Helm release with the default revision. This code invokes helm get manifest without --revision (cmd/helm.go:139-158), so Helm must list the active Secret/ConfigMap storage backend to find the newest revision; granting only get causes the diff to fail before resource reads. Grant list to release storage while keeping workload kinds at get.
 - apiGroups: [""]
resources: ["configmaps", "secrets", "services", "serviceaccounts"]
verbs: ["get"]
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment threadmanifest/generate.go
A MethodNotAllowed no longer aborts. The authorization review added in
the previous commit ran for every refusal isPatchNotAllowed accepts, so a
405 against credentials that may patch was read as "not a permission
problem" and surfaced. A 405 says the API server takes no patch for the
resource whoever is asking, so being entitled to send one changes
nothing; only a 403 is ambiguous enough to be worth the review.
The least-privilege role could not read the release. Without an explicit
--revision helm lists the storage backend to find the newest one, so
`get` on Secrets and ConfigMaps is not enough - the release storage needs
`list` as well, while the kinds a chart renders still only need `get`.
The HELM_DIFF_THREE_WAY_MERGE_MODE examples did not enable the three-way
merge, so they demonstrated an ordinary two-way diff. Now that the
variable is deliberately ignored unless the merge is on, they have to
pass --three-way-merge to show anything. The README copy of the example
block is maintained by hand rather than by scripts/gen-readme.sh, which
only regenerates the flag tables, so it had drifted from the binary.
The README also documents that client mode can be quieter than the truth
in one place: the restoration copies back every live-only field a
wholesale replacement removed, and nothing local can tell a value the API
server defaulted from one somebody set by hand. A NetworkPolicy port with
a hand-added endPort keeps it, where the upgrade would drop it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oreonl

Copy link
Copy Markdown
Author

All five points from this review are addressed in d8d73ff. The four suppressed ones have no threads, so answering them here.

MethodNotAllowed aborting instead of falling back (manifest/generate.go:233) — correct, and a regression from the SelfSubjectAccessReview I added in ca059ad: the review ran for every refusal isPatchNotAllowed accepts, so a 405 against credentials that may patch was surfaced rather than falling back. Reproduced (patchDenied(405, canPatch=true) returned false) and fixed by restricting the disambiguation to Forbidden — a 405 says the API server takes no patch for that resource whoever is asking, so entitlement changes nothing. Covered by TestClientSideFallbackAlwaysFallsBackOnMethodNotAllowed.

Release storage needs list (README.md:276, :297, :299) — confirmed in the code: helmGetArgs omits --revision when the revision is 0 (cmd/helm.go:141-145), and helm's Secrets driver Query does a List to find the newest revision, so get alone fails before any resource is read. The least-privilege example now grants get, list on configmaps/secrets for the release storage and keeps the rendered kinds at get, with the reason spelled out in the surrounding prose.

The env var examples did not enable the three-way merge (README.md:404, cmd/upgrade.go:170) — right, and self-inflicted: making HELM_DIFF_THREE_WAY_MERGE_MODE deliberately ignored unless the merge is on invalidated my own examples. Both now pass --three-way-merge. Worth flagging for future edits that scripts/gen-readme.sh regenerates only the flag tables, so the README's copy of the example block is hand-maintained and had drifted from the binary; I diffed the two to confirm they match again.

The remaining inline comment, about live-only drift inside a replaced atomic list, is answered on its own thread. I have left that one open, since it is a design trade-off rather than something I have fixed.

For what it is worth while the checks are still pending: golangci-lint v2.12.2 against Go 1.26.4 — the pairing lint.yaml pins via go-version-file — reports 0 issues, and go test ./... passes.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Local normalization can discard newer Kubernetes fields, while explicit-null restoration can hide real drift corrections.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

manifest/generate.go:365

  • Decoding into the compiled Go type silently discards fields unknown to this client-go version. If helm-diff runs against a newer cluster and a chart adds a newly supported field to an otherwise known built-in kind, the local merge contains the field here, this round-trip removes it, and the restoration pass cannot recover it because it is absent from the live object. Client mode then omits a real upgrade change; preserve unknown fields from the merged object during normalization or avoid typed normalization when the typed schema does not cover the object.

manifest/generate.go:474

  • This treats an explicit null exactly like a missing key, but null can be the chart-owned desired value for fields that are not defaulted. For example, if both manifests render runtimeClassName: null while the live Pod template drifted to runtimeClassName: gvisor, the three-way patch clears the field and normalize omits it; this condition then restores gvisor, hiding a correction that the real upgrade makes. Use presence-aware lookups and restrict null restoration to fields known to be server-defaulted rather than restoring every explicit-null field.
 if originalMap[key] == nil && modifiedMap[key] == nil {
mergedMap[key] = liveValue
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@oreonl@yxxhero
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat: make --three-way-merge work without patch permissions - #1063

Open
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission
Open

feat: make --three-way-merge work without patch permissions#1063
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission

Conversation

@oreonl

Copy link
Copy Markdown

What

--three-way-merge currently requires the patch permission on every diffed resource. It computes the merge patch locally but asks the API server to apply it as a dry-run:

helper.ServerDryRun=truetargetObj, err:=helper.Patch(info.Namespace, info.Name, patchType, patch, nil)

A read-only account gets cannot patch "x" with kind Deployment: ... is forbidden, so the feature is unavailable to exactly the credentials a "show me what this upgrade would do" tool tends to run under.

This applies the patch locally when the server round-trip is not available, using the same strategic-merge (or JSON merge patch, for custom resources) logic the API server would use. Only get is required.

Interface

A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE):

modebehaviour
auto (default)server dry-run, falling back to the local merge on Forbidden / MethodNotAllowed. Any other error still aborts, so a genuinely bad patch is not masked.
serverthe previous behaviour — a missing patch permission is a hard error.
clientnever sends the patch at all.

auto means a read-only account works out of the box; it only degrades when the accurate path is unavailable anyway, and says so on stderr.

manifest.Generate takes variadic options rather than a new parameter, so existing callers (helmfile vendors this package) keep compiling.

Matching what the server returns

A naive local merge produces a diff full of noise, because the API server does more than apply the patch. scheme.Scheme.Default() is a no-op in client-go — the SetDefaults_* functions live in k8s.io/kubernetes and are not importable — so the defaults cannot simply be recomputed. Two post-processing steps recover most of the gap:

1. Round-trip through the Go type, the way the API server does before it answers. That drops the empty values a manifest spells out but the type omits — initialDelaySeconds: 0, hostNetwork: false, sysctls: [], supplementalGroups: [] — which otherwise appear as additions the upgrade does not make. (Helm's kube.Client.Build returns unstructured objects, so these survive from the chart YAML verbatim.)

2. Copy a field back from the live object when the old and the new release manifest agree about it. Values disappear from the merged object two ways:

  • the patch replaces retainKeys structs (spec.strategy) and atomic lists (ports, volumeClaimTemplates) as a whole, dropping what the server had defaulted into them;
  • a manifest that renders an unset value as an explicit null — a bare replicas:, which is what a chart writes for a value it leaves unset — puts the key in both manifests, so it reaches the patch as a change rather than a deletion. That deletes a defaulted value even when the chart did not change at all.

The server re-defaults both cases immediately after applying the patch. Locally, a field is restored only when nothing in either manifest asked for it to go. Where the two manifests disagree the deletion is honoured, because that is a change the chart really makes — restoring there would hide real removals (a dropped nodeSelector entry looks identical at the JSON level, and the server does not default that back). Values already present in the merged object are never overwritten, so drift between the cluster and the chart is still reported and still corrected.

Known deviation from server mode

When the manifests genuinely disagree about a field the API server defaults, client mode reports it as removed where server mode shows it changing to the default — a chart that stops pinning replicas: 3 shows the field going away rather than 31. The change is reported either way; client mode just cannot name the value that replaces it. Mutating webhooks and validation are also not applied. All of this is documented in the new README section, along with the minimal RBAC role.

Tests

manifest/generate_test.go is new. Eight TestLocalMerge_* cases are built from real manifests (unstructured, as kube.Client.Build supplies them; the live side written as the API server returns it) and cover each symptom above, plus:

  • a chart that genuinely stops setting a field — still reported as removed;
  • a hand-scaled replicas and hand-edited image in the cluster — still reset to the chart values, i.e. drift detection intact.

I checked the tests are not vacuous by reverting each fix independently; each fails without the step it covers. cmd/upgrade_test.go adds flag/env validation cases.

go test ./..., go vet and gofmt pass, and the README flag tables are regenerated with make readme.

Testing done

Built with make install/helm and run against a real cluster with an account lacking patch, on a release of ~15 resources (LibreChat and its subcharts). Before the two post-processing steps the client-side output differed from --three-way-merge-mode=server on defaulted fields; the cases in the tests are taken from that output.

🤖 Generated with Claude Code

oreonland others added 2 commits August 31, 2026 11:23
--three-way-merge computed the merge patch locally but asked the API
server to apply it as a dry-run, which needs the `patch` verb on every
diffed resource. A read-only account got:
cannot patch "x" with kind Deployment: ... is forbidden
The patch is now applied locally when the server round-trip is not
available, using the same strategic-merge (or JSON merge patch, for
custom resources) logic the API server would use. Only `get` is
required.
A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE)
selects between:
auto server dry-run, falling back to the local merge on Forbidden
or MethodNotAllowed. Any other error still aborts, so a
genuinely bad patch is not masked. The default.
server the previous behaviour; a missing permission is a hard error.
client never sends the patch at all.
Two post-processing steps keep the local result close to what the API
server returns, since client-go does not ship the defaulting functions:
* The merged object is round-tripped through its Go type, the way the
API server does before it answers. That drops the empty values a
manifest spells out but the type omits - `initialDelaySeconds: 0`,
`hostNetwork: false`, `sysctls: []` - which would otherwise show up
as additions the upgrade does not make.
* A field is copied back from the live object when the old and the new
release manifest agree about it. The patch replaces `retainKeys`
structs and atomic lists as a whole, and a manifest that renders an
unset value as an explicit `null` (a bare `replicas:`) reaches the
patch as a change rather than a deletion; both wipe values the API
server had defaulted in and re-defaults immediately after. Fields
the two manifests disagree about are left deleted, because that is a
change the chart really makes.
manifest.Generate takes variadic options rather than a new parameter, so
existing callers keep compiling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A chart that renders an empty collection literally - `rules: []`, the
branch grafana's Role and ClusterRole take when no sidecar is enabled -
disagrees with the cluster about how to write "nothing". Kubernetes
stores objects as protobuf, which cannot tell an empty repeated field
from an absent one, so the API server answers with `rules: null`.
The patch then carries `{"rules":[]}` even though the chart did not
change, and the client-side merge reported:
rules (rbac.authorization.k8s.io/v1/Role/grafana/grafana)
± type change from <nil> to list
server mode shows nothing, because its answer goes back through the same
storage round-trip. The local merge now keeps whichever spelling the
cluster reports whenever the merged and the live value are both empty.
Emptying a collection that actually had entries is still a change and is
still reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

auto mode does not currently “fall back per run” (it keeps attempting forbidden PATCHes) and the docs/RBAC wording plus env-var handling need small corrections to match intended behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new three-way-merge “mode” capability so helm diff upgrade --three-way-merge can work with read-only Kubernetes credentials by falling back (or opting in) to a client-side patch application that approximates the API server’s results.

Changes:

  • Introduces --three-way-merge-mode (auto/server/client) and env HELM_DIFF_THREE_WAY_MERGE_MODE, wiring it through cmd/upgrade into manifest.Generate(...) via variadic options.
  • Implements client-side patch application for three-way-merge, including normalization (round-trip through Go types) and a restoration pass for server-populated/defaulted fields.
  • Adds focused unit tests for local merge behavior and CLI flag/env validation; updates README and Copilot instructions.
File summaries
FileDescription
README.mdDocuments --three-way-merge-mode, behavior tradeoffs, and RBAC guidance.
manifest/generate.goAdds merge-mode options and client-side patch application with post-processing to reduce diff noise.
manifest/generate_test.goNew tests covering strategic/merge patch local application and default-restoration behaviors.
cmd/upgrade.goAdds flag/env plumbing and validation; passes merge-mode into manifest generation.
cmd/upgrade_test.goAdds validation coverage for the new flag/env behavior.
.github/copilot-instructions.mdDocuments the new environment variable.
Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadmanifest/generate.go Outdated
Comment threadcmd/upgrade.go Outdated
Comment threadREADME.md Outdated
Comment threadmanifest/generate.go Outdated
auto mode now falls back for the whole run, as documented. It warned
once but kept sending a dry-run PATCH for every remaining resource, so a
release of N objects produced N denied requests - N audit log entries and
N round-trips - to reach the same conclusion each time. A refused dry-run
says something about the credentials rather than about the resource, so
the first refusal now moves the rest of the run to the local merge. The
decision moved into a clientSideFallback type so it can be tested without
a cluster.
HELM_DIFF_THREE_WAY_MERGE_MODE is read only when the run actually
performs a three-way merge (--three-way-merge, or --take-ownership which
implies it). A stale value in the environment could otherwise fail an
unrelated `helm diff upgrade` over a mode it was never going to use. The
flag stays validated unconditionally: passing it is deliberate, so a typo
is worth reporting either way.
The README no longer calls a wildcard role "minimal RBAC" - it grants
read access to every resource in every API group. It is now described as
the blunt option, next to a scoped example and a note that a kind missing
from the role fails the diff for that resource.
Also fixes the grammar in the two createPatch comments the review
flagged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new validation error messages format the allowed-mode slice with %q, which will render as %!q([]string=...) and should be corrected before merge.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

cmd/upgrade.go:199

  • The format string uses %q for manifest.ValidThreeWayMergeModes (a []string), which will render as %!q([]string=...) and produce an unhelpful validation error. Use %v (or join the slice) when printing the allowed values.

This issue also appears on line 226 of the same file.

cmd/upgrade.go:228

  • The env-var validation error formats manifest.ValidThreeWayMergeModes (a []string) with %q, which will print as %!q([]string=...). Use %v (or join the slice) so users see the allowed values clearly.
 if !slices.Contains(manifest.ValidThreeWayMergeModes, mode) {
return fmt.Errorf("env var %q must be one of %q, but got %q", "HELM_DIFF_THREE_WAY_MERGE_MODE", manifest.ValidThreeWayMergeModes, mode)
}
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@oreonl

Copy link
Copy Markdown
Author

Checked this one and I don't think it holds — %q on a []string isn't malformed. fmt applies a verb recursively to the elements of a compound operand (docs: "For compound objects, the elements are printed using these rules, recursively... array, slice: [elem0 elem1 ...]"). %!q(...) only appears when the verb is invalid for the type, e.g. %q on a struct.

Actual output from the current branch, for both lines flagged:

$ helm diff upgrade rel ./chart --three-way-merge-mode local
Error: flag "three-way-merge-mode" must be one of ["auto" "server" "client"], but got "local"
$ HELM_DIFF_THREE_WAY_MERGE_MODE=local helm diff upgrade rel ./chart --three-way-merge
Error: env var "HELM_DIFF_THREE_WAY_MERGE_MODE" must be one of ["auto" "server" "client"], but got "local"

%v would print [auto server client], dropping the quotes that delimit the values — and the adjacent --server-side check quotes each one:

Error: flag "server-side" must be "true", "false" or "auto", but got "yes"

So I've left these as they are.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core diff computation behavior with non-trivial Kubernetes patch/defaulting semantics, so it warrants final human review despite good test coverage.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment threadmanifest/generate.go Outdated
The marshal of originalObj reported "serializing current configuration",
while the live object marshalled a few lines below reported "serializing
live configuration". Two different objects, and the first was named after
the second, which is misleading when a failure has to be diagnosed. The
three messages now say original, target and live.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core three-way-merge behavior with substantial new merge/restoration logic whose correctness and compatibility impact are difficult to fully validate via automated review alone.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@yxxhero

Copy link
Copy Markdown
Collaborator

@oreonl PTAL

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Local restoration can hide real drift and CRD changes, while automatic fallback can mask admission failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadREADME.md Outdated
oreonland others added 2 commits September 3, 2026 13:55
Four defects from review, all confirmed by reproduction before the fix.
restoreMissing copied a live value back whenever the two release
manifests agreed about a key, including when they agreed on an actual
value. normalize drops an explicit `hostNetwork: false` through
omitempty, so a chart pinning false against a cluster that drifted to
true had the drift restored and the correction vanished from the diff -
the opposite of what three-way merge is for. A value is now restored only
where neither manifest carries one, an absent key and a `null` counting
alike, which is all the unset `replicas:` case ever needed.
The restoration pass no longer runs behind a JSON merge patch. That patch
carries only what the chart changed between the two manifests, so it
never prunes a field the cluster populated and there is nothing to put
back. Custom resources are also stored as JSON rather than protobuf,
which keeps `null`, `[]` and `{}` apart, so the empty-value equivalence
was undoing a chart deliberately changing one into another.
auto no longer reads every 403 as a missing patch permission. An
admission webhook or a quota controller can refuse a request the
credentials are entitled to make, and that refusal is exactly what the
upgrade would hit, so falling back locally hid the outcome the diff
exists to predict. A SelfSubjectAccessReview now settles whether patching
is permitted before the refusal is treated as a permission problem; where
the review cannot be made the previous behaviour stands. The fallback is
already once per run, so this costs one extra request.
The README documents a second deviation from server mode: changing an
entry inside an atomic list also drops the server defaults elsewhere in
that entry, because positions in a list the chart rewrote cannot be
matched up safely.
k8s.io/api/authorization/v1 is added to the depguard allowlist, which
golangci-lint requires for the new import.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fallback handling and field restoration have correctness gaps, and several usage and RBAC examples are inaccurate.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (5)

Previously missed (3) — in code that hasn't changed since the last review.

README.md:276

  • The release-storage requirement is understated here: the default revision is resolved by helm get manifest without --revision, which lists the Helm storage backend to find the newest release. A role with only get on Secrets/ConfigMaps therefore fails for the normal invocation; document list as required as well.

This issue also appears on line 297 of the same file.
README.md:404

  • This generated example does not enable three-way merge. Since the mode environment variable is intentionally ignored unless three-way merge is already active, the command performs a normal two-way diff and does not demonstrate client-side merging.
    cmd/upgrade.go:170
  • This example does not enable three-way merge. The command only reads this environment variable when HELM_DIFF_THREE_WAY_MERGE=true (or --three-way-merge/--take-ownership) is already active, so as written it performs the ordinary two-way diff rather than a local merge.

manifest/generate.go:233

  • MethodNotAllowed never falls back when the authorization review says patching is allowed: isPatchNotAllowed routes it here, then this check returns false and aborts. That contradicts the new auto contract, which explicitly falls back for MethodNotAllowed. Restrict the authorization disambiguation to Forbidden; a 405 indicates the server path is unavailable regardless of RBAC.
	if f.canPatch != nil {
if allowed, err := f.canPatch(info); err == nil && allowed {
return false
}

README.md:299

  • The least-privilege example cannot read the latest Helm release with the default revision. This code invokes helm get manifest without --revision (cmd/helm.go:139-158), so Helm must list the active Secret/ConfigMap storage backend to find the newest revision; granting only get causes the diff to fail before resource reads. Grant list to release storage while keeping workload kinds at get.
 - apiGroups: [""]
resources: ["configmaps", "secrets", "services", "serviceaccounts"]
verbs: ["get"]
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment threadmanifest/generate.go
A MethodNotAllowed no longer aborts. The authorization review added in
the previous commit ran for every refusal isPatchNotAllowed accepts, so a
405 against credentials that may patch was read as "not a permission
problem" and surfaced. A 405 says the API server takes no patch for the
resource whoever is asking, so being entitled to send one changes
nothing; only a 403 is ambiguous enough to be worth the review.
The least-privilege role could not read the release. Without an explicit
--revision helm lists the storage backend to find the newest one, so
`get` on Secrets and ConfigMaps is not enough - the release storage needs
`list` as well, while the kinds a chart renders still only need `get`.
The HELM_DIFF_THREE_WAY_MERGE_MODE examples did not enable the three-way
merge, so they demonstrated an ordinary two-way diff. Now that the
variable is deliberately ignored unless the merge is on, they have to
pass --three-way-merge to show anything. The README copy of the example
block is maintained by hand rather than by scripts/gen-readme.sh, which
only regenerates the flag tables, so it had drifted from the binary.
The README also documents that client mode can be quieter than the truth
in one place: the restoration copies back every live-only field a
wholesale replacement removed, and nothing local can tell a value the API
server defaulted from one somebody set by hand. A NetworkPolicy port with
a hand-added endPort keeps it, where the upgrade would drop it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oreonl

Copy link
Copy Markdown
Author

All five points from this review are addressed in d8d73ff. The four suppressed ones have no threads, so answering them here.

MethodNotAllowed aborting instead of falling back (manifest/generate.go:233) — correct, and a regression from the SelfSubjectAccessReview I added in ca059ad: the review ran for every refusal isPatchNotAllowed accepts, so a 405 against credentials that may patch was surfaced rather than falling back. Reproduced (patchDenied(405, canPatch=true) returned false) and fixed by restricting the disambiguation to Forbidden — a 405 says the API server takes no patch for that resource whoever is asking, so entitlement changes nothing. Covered by TestClientSideFallbackAlwaysFallsBackOnMethodNotAllowed.

Release storage needs list (README.md:276, :297, :299) — confirmed in the code: helmGetArgs omits --revision when the revision is 0 (cmd/helm.go:141-145), and helm's Secrets driver Query does a List to find the newest revision, so get alone fails before any resource is read. The least-privilege example now grants get, list on configmaps/secrets for the release storage and keeps the rendered kinds at get, with the reason spelled out in the surrounding prose.

The env var examples did not enable the three-way merge (README.md:404, cmd/upgrade.go:170) — right, and self-inflicted: making HELM_DIFF_THREE_WAY_MERGE_MODE deliberately ignored unless the merge is on invalidated my own examples. Both now pass --three-way-merge. Worth flagging for future edits that scripts/gen-readme.sh regenerates only the flag tables, so the README's copy of the example block is hand-maintained and had drifted from the binary; I diffed the two to confirm they match again.

The remaining inline comment, about live-only drift inside a replaced atomic list, is answered on its own thread. I have left that one open, since it is a design trade-off rather than something I have fixed.

For what it is worth while the checks are still pending: golangci-lint v2.12.2 against Go 1.26.4 — the pairing lint.yaml pins via go-version-file — reports 0 issues, and go test ./... passes.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Local normalization can discard newer Kubernetes fields, while explicit-null restoration can hide real drift corrections.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

manifest/generate.go:365

  • Decoding into the compiled Go type silently discards fields unknown to this client-go version. If helm-diff runs against a newer cluster and a chart adds a newly supported field to an otherwise known built-in kind, the local merge contains the field here, this round-trip removes it, and the restoration pass cannot recover it because it is absent from the live object. Client mode then omits a real upgrade change; preserve unknown fields from the merged object during normalization or avoid typed normalization when the typed schema does not cover the object.

manifest/generate.go:474

  • This treats an explicit null exactly like a missing key, but null can be the chart-owned desired value for fields that are not defaulted. For example, if both manifests render runtimeClassName: null while the live Pod template drifted to runtimeClassName: gvisor, the three-way patch clears the field and normalize omits it; this condition then restores gvisor, hiding a correction that the real upgrade makes. Use presence-aware lookups and restrict null restoration to fields known to be server-defaulted rather than restoring every explicit-null field.
 if originalMap[key] == nil && modifiedMap[key] == nil {
mergedMap[key] = liveValue
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@oreonl@yxxhero
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

feat: make --three-way-merge work without patch permissions - #1063

Open
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission
Open

feat: make --three-way-merge work without patch permissions#1063
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission

Conversation

@oreonl

Copy link
Copy Markdown

What

--three-way-merge currently requires the patch permission on every diffed resource. It computes the merge patch locally but asks the API server to apply it as a dry-run:

helper.ServerDryRun=truetargetObj, err:=helper.Patch(info.Namespace, info.Name, patchType, patch, nil)

A read-only account gets cannot patch "x" with kind Deployment: ... is forbidden, so the feature is unavailable to exactly the credentials a "show me what this upgrade would do" tool tends to run under.

This applies the patch locally when the server round-trip is not available, using the same strategic-merge (or JSON merge patch, for custom resources) logic the API server would use. Only get is required.

Interface

A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE):

modebehaviour
auto (default)server dry-run, falling back to the local merge on Forbidden / MethodNotAllowed. Any other error still aborts, so a genuinely bad patch is not masked.
serverthe previous behaviour — a missing patch permission is a hard error.
clientnever sends the patch at all.

auto means a read-only account works out of the box; it only degrades when the accurate path is unavailable anyway, and says so on stderr.

manifest.Generate takes variadic options rather than a new parameter, so existing callers (helmfile vendors this package) keep compiling.

Matching what the server returns

A naive local merge produces a diff full of noise, because the API server does more than apply the patch. scheme.Scheme.Default() is a no-op in client-go — the SetDefaults_* functions live in k8s.io/kubernetes and are not importable — so the defaults cannot simply be recomputed. Two post-processing steps recover most of the gap:

1. Round-trip through the Go type, the way the API server does before it answers. That drops the empty values a manifest spells out but the type omits — initialDelaySeconds: 0, hostNetwork: false, sysctls: [], supplementalGroups: [] — which otherwise appear as additions the upgrade does not make. (Helm's kube.Client.Build returns unstructured objects, so these survive from the chart YAML verbatim.)

2. Copy a field back from the live object when the old and the new release manifest agree about it. Values disappear from the merged object two ways:

  • the patch replaces retainKeys structs (spec.strategy) and atomic lists (ports, volumeClaimTemplates) as a whole, dropping what the server had defaulted into them;
  • a manifest that renders an unset value as an explicit null — a bare replicas:, which is what a chart writes for a value it leaves unset — puts the key in both manifests, so it reaches the patch as a change rather than a deletion. That deletes a defaulted value even when the chart did not change at all.

The server re-defaults both cases immediately after applying the patch. Locally, a field is restored only when nothing in either manifest asked for it to go. Where the two manifests disagree the deletion is honoured, because that is a change the chart really makes — restoring there would hide real removals (a dropped nodeSelector entry looks identical at the JSON level, and the server does not default that back). Values already present in the merged object are never overwritten, so drift between the cluster and the chart is still reported and still corrected.

Known deviation from server mode

When the manifests genuinely disagree about a field the API server defaults, client mode reports it as removed where server mode shows it changing to the default — a chart that stops pinning replicas: 3 shows the field going away rather than 31. The change is reported either way; client mode just cannot name the value that replaces it. Mutating webhooks and validation are also not applied. All of this is documented in the new README section, along with the minimal RBAC role.

Tests

manifest/generate_test.go is new. Eight TestLocalMerge_* cases are built from real manifests (unstructured, as kube.Client.Build supplies them; the live side written as the API server returns it) and cover each symptom above, plus:

  • a chart that genuinely stops setting a field — still reported as removed;
  • a hand-scaled replicas and hand-edited image in the cluster — still reset to the chart values, i.e. drift detection intact.

I checked the tests are not vacuous by reverting each fix independently; each fails without the step it covers. cmd/upgrade_test.go adds flag/env validation cases.

go test ./..., go vet and gofmt pass, and the README flag tables are regenerated with make readme.

Testing done

Built with make install/helm and run against a real cluster with an account lacking patch, on a release of ~15 resources (LibreChat and its subcharts). Before the two post-processing steps the client-side output differed from --three-way-merge-mode=server on defaulted fields; the cases in the tests are taken from that output.

🤖 Generated with Claude Code

oreonland others added 2 commits August 31, 2026 11:23
--three-way-merge computed the merge patch locally but asked the API
server to apply it as a dry-run, which needs the `patch` verb on every
diffed resource. A read-only account got:
cannot patch "x" with kind Deployment: ... is forbidden
The patch is now applied locally when the server round-trip is not
available, using the same strategic-merge (or JSON merge patch, for
custom resources) logic the API server would use. Only `get` is
required.
A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE)
selects between:
auto server dry-run, falling back to the local merge on Forbidden
or MethodNotAllowed. Any other error still aborts, so a
genuinely bad patch is not masked. The default.
server the previous behaviour; a missing permission is a hard error.
client never sends the patch at all.
Two post-processing steps keep the local result close to what the API
server returns, since client-go does not ship the defaulting functions:
* The merged object is round-tripped through its Go type, the way the
API server does before it answers. That drops the empty values a
manifest spells out but the type omits - `initialDelaySeconds: 0`,
`hostNetwork: false`, `sysctls: []` - which would otherwise show up
as additions the upgrade does not make.
* A field is copied back from the live object when the old and the new
release manifest agree about it. The patch replaces `retainKeys`
structs and atomic lists as a whole, and a manifest that renders an
unset value as an explicit `null` (a bare `replicas:`) reaches the
patch as a change rather than a deletion; both wipe values the API
server had defaulted in and re-defaults immediately after. Fields
the two manifests disagree about are left deleted, because that is a
change the chart really makes.
manifest.Generate takes variadic options rather than a new parameter, so
existing callers keep compiling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A chart that renders an empty collection literally - `rules: []`, the
branch grafana's Role and ClusterRole take when no sidecar is enabled -
disagrees with the cluster about how to write "nothing". Kubernetes
stores objects as protobuf, which cannot tell an empty repeated field
from an absent one, so the API server answers with `rules: null`.
The patch then carries `{"rules":[]}` even though the chart did not
change, and the client-side merge reported:
rules (rbac.authorization.k8s.io/v1/Role/grafana/grafana)
± type change from <nil> to list
server mode shows nothing, because its answer goes back through the same
storage round-trip. The local merge now keeps whichever spelling the
cluster reports whenever the merged and the live value are both empty.
Emptying a collection that actually had entries is still a change and is
still reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

auto mode does not currently “fall back per run” (it keeps attempting forbidden PATCHes) and the docs/RBAC wording plus env-var handling need small corrections to match intended behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new three-way-merge “mode” capability so helm diff upgrade --three-way-merge can work with read-only Kubernetes credentials by falling back (or opting in) to a client-side patch application that approximates the API server’s results.

Changes:

  • Introduces --three-way-merge-mode (auto/server/client) and env HELM_DIFF_THREE_WAY_MERGE_MODE, wiring it through cmd/upgrade into manifest.Generate(...) via variadic options.
  • Implements client-side patch application for three-way-merge, including normalization (round-trip through Go types) and a restoration pass for server-populated/defaulted fields.
  • Adds focused unit tests for local merge behavior and CLI flag/env validation; updates README and Copilot instructions.
File summaries
FileDescription
README.mdDocuments --three-way-merge-mode, behavior tradeoffs, and RBAC guidance.
manifest/generate.goAdds merge-mode options and client-side patch application with post-processing to reduce diff noise.
manifest/generate_test.goNew tests covering strategic/merge patch local application and default-restoration behaviors.
cmd/upgrade.goAdds flag/env plumbing and validation; passes merge-mode into manifest generation.
cmd/upgrade_test.goAdds validation coverage for the new flag/env behavior.
.github/copilot-instructions.mdDocuments the new environment variable.
Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadmanifest/generate.go Outdated
Comment threadcmd/upgrade.go Outdated
Comment threadREADME.md Outdated
Comment threadmanifest/generate.go Outdated
auto mode now falls back for the whole run, as documented. It warned
once but kept sending a dry-run PATCH for every remaining resource, so a
release of N objects produced N denied requests - N audit log entries and
N round-trips - to reach the same conclusion each time. A refused dry-run
says something about the credentials rather than about the resource, so
the first refusal now moves the rest of the run to the local merge. The
decision moved into a clientSideFallback type so it can be tested without
a cluster.
HELM_DIFF_THREE_WAY_MERGE_MODE is read only when the run actually
performs a three-way merge (--three-way-merge, or --take-ownership which
implies it). A stale value in the environment could otherwise fail an
unrelated `helm diff upgrade` over a mode it was never going to use. The
flag stays validated unconditionally: passing it is deliberate, so a typo
is worth reporting either way.
The README no longer calls a wildcard role "minimal RBAC" - it grants
read access to every resource in every API group. It is now described as
the blunt option, next to a scoped example and a note that a kind missing
from the role fails the diff for that resource.
Also fixes the grammar in the two createPatch comments the review
flagged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new validation error messages format the allowed-mode slice with %q, which will render as %!q([]string=...) and should be corrected before merge.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

cmd/upgrade.go:199

  • The format string uses %q for manifest.ValidThreeWayMergeModes (a []string), which will render as %!q([]string=...) and produce an unhelpful validation error. Use %v (or join the slice) when printing the allowed values.

This issue also appears on line 226 of the same file.

cmd/upgrade.go:228

  • The env-var validation error formats manifest.ValidThreeWayMergeModes (a []string) with %q, which will print as %!q([]string=...). Use %v (or join the slice) so users see the allowed values clearly.
 if !slices.Contains(manifest.ValidThreeWayMergeModes, mode) {
return fmt.Errorf("env var %q must be one of %q, but got %q", "HELM_DIFF_THREE_WAY_MERGE_MODE", manifest.ValidThreeWayMergeModes, mode)
}
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@oreonl

Copy link
Copy Markdown
Author

Checked this one and I don't think it holds — %q on a []string isn't malformed. fmt applies a verb recursively to the elements of a compound operand (docs: "For compound objects, the elements are printed using these rules, recursively... array, slice: [elem0 elem1 ...]"). %!q(...) only appears when the verb is invalid for the type, e.g. %q on a struct.

Actual output from the current branch, for both lines flagged:

$ helm diff upgrade rel ./chart --three-way-merge-mode local
Error: flag "three-way-merge-mode" must be one of ["auto" "server" "client"], but got "local"
$ HELM_DIFF_THREE_WAY_MERGE_MODE=local helm diff upgrade rel ./chart --three-way-merge
Error: env var "HELM_DIFF_THREE_WAY_MERGE_MODE" must be one of ["auto" "server" "client"], but got "local"

%v would print [auto server client], dropping the quotes that delimit the values — and the adjacent --server-side check quotes each one:

Error: flag "server-side" must be "true", "false" or "auto", but got "yes"

So I've left these as they are.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core diff computation behavior with non-trivial Kubernetes patch/defaulting semantics, so it warrants final human review despite good test coverage.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment threadmanifest/generate.go Outdated
The marshal of originalObj reported "serializing current configuration",
while the live object marshalled a few lines below reported "serializing
live configuration". Two different objects, and the first was named after
the second, which is misleading when a failure has to be diagnosed. The
three messages now say original, target and live.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core three-way-merge behavior with substantial new merge/restoration logic whose correctness and compatibility impact are difficult to fully validate via automated review alone.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@yxxhero

Copy link
Copy Markdown
Collaborator

@oreonl PTAL

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Local restoration can hide real drift and CRD changes, while automatic fallback can mask admission failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadREADME.md Outdated
oreonland others added 2 commits September 3, 2026 13:55
Four defects from review, all confirmed by reproduction before the fix.
restoreMissing copied a live value back whenever the two release
manifests agreed about a key, including when they agreed on an actual
value. normalize drops an explicit `hostNetwork: false` through
omitempty, so a chart pinning false against a cluster that drifted to
true had the drift restored and the correction vanished from the diff -
the opposite of what three-way merge is for. A value is now restored only
where neither manifest carries one, an absent key and a `null` counting
alike, which is all the unset `replicas:` case ever needed.
The restoration pass no longer runs behind a JSON merge patch. That patch
carries only what the chart changed between the two manifests, so it
never prunes a field the cluster populated and there is nothing to put
back. Custom resources are also stored as JSON rather than protobuf,
which keeps `null`, `[]` and `{}` apart, so the empty-value equivalence
was undoing a chart deliberately changing one into another.
auto no longer reads every 403 as a missing patch permission. An
admission webhook or a quota controller can refuse a request the
credentials are entitled to make, and that refusal is exactly what the
upgrade would hit, so falling back locally hid the outcome the diff
exists to predict. A SelfSubjectAccessReview now settles whether patching
is permitted before the refusal is treated as a permission problem; where
the review cannot be made the previous behaviour stands. The fallback is
already once per run, so this costs one extra request.
The README documents a second deviation from server mode: changing an
entry inside an atomic list also drops the server defaults elsewhere in
that entry, because positions in a list the chart rewrote cannot be
matched up safely.
k8s.io/api/authorization/v1 is added to the depguard allowlist, which
golangci-lint requires for the new import.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fallback handling and field restoration have correctness gaps, and several usage and RBAC examples are inaccurate.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (5)

Previously missed (3) — in code that hasn't changed since the last review.

README.md:276

  • The release-storage requirement is understated here: the default revision is resolved by helm get manifest without --revision, which lists the Helm storage backend to find the newest release. A role with only get on Secrets/ConfigMaps therefore fails for the normal invocation; document list as required as well.

This issue also appears on line 297 of the same file.
README.md:404

  • This generated example does not enable three-way merge. Since the mode environment variable is intentionally ignored unless three-way merge is already active, the command performs a normal two-way diff and does not demonstrate client-side merging.
    cmd/upgrade.go:170
  • This example does not enable three-way merge. The command only reads this environment variable when HELM_DIFF_THREE_WAY_MERGE=true (or --three-way-merge/--take-ownership) is already active, so as written it performs the ordinary two-way diff rather than a local merge.

manifest/generate.go:233

  • MethodNotAllowed never falls back when the authorization review says patching is allowed: isPatchNotAllowed routes it here, then this check returns false and aborts. That contradicts the new auto contract, which explicitly falls back for MethodNotAllowed. Restrict the authorization disambiguation to Forbidden; a 405 indicates the server path is unavailable regardless of RBAC.
	if f.canPatch != nil {
if allowed, err := f.canPatch(info); err == nil && allowed {
return false
}

README.md:299

  • The least-privilege example cannot read the latest Helm release with the default revision. This code invokes helm get manifest without --revision (cmd/helm.go:139-158), so Helm must list the active Secret/ConfigMap storage backend to find the newest revision; granting only get causes the diff to fail before resource reads. Grant list to release storage while keeping workload kinds at get.
 - apiGroups: [""]
resources: ["configmaps", "secrets", "services", "serviceaccounts"]
verbs: ["get"]
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment threadmanifest/generate.go
A MethodNotAllowed no longer aborts. The authorization review added in
the previous commit ran for every refusal isPatchNotAllowed accepts, so a
405 against credentials that may patch was read as "not a permission
problem" and surfaced. A 405 says the API server takes no patch for the
resource whoever is asking, so being entitled to send one changes
nothing; only a 403 is ambiguous enough to be worth the review.
The least-privilege role could not read the release. Without an explicit
--revision helm lists the storage backend to find the newest one, so
`get` on Secrets and ConfigMaps is not enough - the release storage needs
`list` as well, while the kinds a chart renders still only need `get`.
The HELM_DIFF_THREE_WAY_MERGE_MODE examples did not enable the three-way
merge, so they demonstrated an ordinary two-way diff. Now that the
variable is deliberately ignored unless the merge is on, they have to
pass --three-way-merge to show anything. The README copy of the example
block is maintained by hand rather than by scripts/gen-readme.sh, which
only regenerates the flag tables, so it had drifted from the binary.
The README also documents that client mode can be quieter than the truth
in one place: the restoration copies back every live-only field a
wholesale replacement removed, and nothing local can tell a value the API
server defaulted from one somebody set by hand. A NetworkPolicy port with
a hand-added endPort keeps it, where the upgrade would drop it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oreonl

Copy link
Copy Markdown
Author

All five points from this review are addressed in d8d73ff. The four suppressed ones have no threads, so answering them here.

MethodNotAllowed aborting instead of falling back (manifest/generate.go:233) — correct, and a regression from the SelfSubjectAccessReview I added in ca059ad: the review ran for every refusal isPatchNotAllowed accepts, so a 405 against credentials that may patch was surfaced rather than falling back. Reproduced (patchDenied(405, canPatch=true) returned false) and fixed by restricting the disambiguation to Forbidden — a 405 says the API server takes no patch for that resource whoever is asking, so entitlement changes nothing. Covered by TestClientSideFallbackAlwaysFallsBackOnMethodNotAllowed.

Release storage needs list (README.md:276, :297, :299) — confirmed in the code: helmGetArgs omits --revision when the revision is 0 (cmd/helm.go:141-145), and helm's Secrets driver Query does a List to find the newest revision, so get alone fails before any resource is read. The least-privilege example now grants get, list on configmaps/secrets for the release storage and keeps the rendered kinds at get, with the reason spelled out in the surrounding prose.

The env var examples did not enable the three-way merge (README.md:404, cmd/upgrade.go:170) — right, and self-inflicted: making HELM_DIFF_THREE_WAY_MERGE_MODE deliberately ignored unless the merge is on invalidated my own examples. Both now pass --three-way-merge. Worth flagging for future edits that scripts/gen-readme.sh regenerates only the flag tables, so the README's copy of the example block is hand-maintained and had drifted from the binary; I diffed the two to confirm they match again.

The remaining inline comment, about live-only drift inside a replaced atomic list, is answered on its own thread. I have left that one open, since it is a design trade-off rather than something I have fixed.

For what it is worth while the checks are still pending: golangci-lint v2.12.2 against Go 1.26.4 — the pairing lint.yaml pins via go-version-file — reports 0 issues, and go test ./... passes.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Local normalization can discard newer Kubernetes fields, while explicit-null restoration can hide real drift corrections.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

manifest/generate.go:365

  • Decoding into the compiled Go type silently discards fields unknown to this client-go version. If helm-diff runs against a newer cluster and a chart adds a newly supported field to an otherwise known built-in kind, the local merge contains the field here, this round-trip removes it, and the restoration pass cannot recover it because it is absent from the live object. Client mode then omits a real upgrade change; preserve unknown fields from the merged object during normalization or avoid typed normalization when the typed schema does not cover the object.

manifest/generate.go:474

  • This treats an explicit null exactly like a missing key, but null can be the chart-owned desired value for fields that are not defaulted. For example, if both manifests render runtimeClassName: null while the live Pod template drifted to runtimeClassName: gvisor, the three-way patch clears the field and normalize omits it; this condition then restores gvisor, hiding a correction that the real upgrade makes. Use presence-aware lookups and restrict null restoration to fields known to be server-defaulted rather than restoring every explicit-null field.
 if originalMap[key] == nil && modifiedMap[key] == nil {
mergedMap[key] = liveValue
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@oreonl@yxxhero
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat: make --three-way-merge work without patch permissions - #1063

Open
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission
Open

feat: make --three-way-merge work without patch permissions#1063
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission

Conversation

@oreonl

Copy link
Copy Markdown

What

--three-way-merge currently requires the patch permission on every diffed resource. It computes the merge patch locally but asks the API server to apply it as a dry-run:

helper.ServerDryRun=truetargetObj, err:=helper.Patch(info.Namespace, info.Name, patchType, patch, nil)

A read-only account gets cannot patch "x" with kind Deployment: ... is forbidden, so the feature is unavailable to exactly the credentials a "show me what this upgrade would do" tool tends to run under.

This applies the patch locally when the server round-trip is not available, using the same strategic-merge (or JSON merge patch, for custom resources) logic the API server would use. Only get is required.

Interface

A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE):

modebehaviour
auto (default)server dry-run, falling back to the local merge on Forbidden / MethodNotAllowed. Any other error still aborts, so a genuinely bad patch is not masked.
serverthe previous behaviour — a missing patch permission is a hard error.
clientnever sends the patch at all.

auto means a read-only account works out of the box; it only degrades when the accurate path is unavailable anyway, and says so on stderr.

manifest.Generate takes variadic options rather than a new parameter, so existing callers (helmfile vendors this package) keep compiling.

Matching what the server returns

A naive local merge produces a diff full of noise, because the API server does more than apply the patch. scheme.Scheme.Default() is a no-op in client-go — the SetDefaults_* functions live in k8s.io/kubernetes and are not importable — so the defaults cannot simply be recomputed. Two post-processing steps recover most of the gap:

1. Round-trip through the Go type, the way the API server does before it answers. That drops the empty values a manifest spells out but the type omits — initialDelaySeconds: 0, hostNetwork: false, sysctls: [], supplementalGroups: [] — which otherwise appear as additions the upgrade does not make. (Helm's kube.Client.Build returns unstructured objects, so these survive from the chart YAML verbatim.)

2. Copy a field back from the live object when the old and the new release manifest agree about it. Values disappear from the merged object two ways:

  • the patch replaces retainKeys structs (spec.strategy) and atomic lists (ports, volumeClaimTemplates) as a whole, dropping what the server had defaulted into them;
  • a manifest that renders an unset value as an explicit null — a bare replicas:, which is what a chart writes for a value it leaves unset — puts the key in both manifests, so it reaches the patch as a change rather than a deletion. That deletes a defaulted value even when the chart did not change at all.

The server re-defaults both cases immediately after applying the patch. Locally, a field is restored only when nothing in either manifest asked for it to go. Where the two manifests disagree the deletion is honoured, because that is a change the chart really makes — restoring there would hide real removals (a dropped nodeSelector entry looks identical at the JSON level, and the server does not default that back). Values already present in the merged object are never overwritten, so drift between the cluster and the chart is still reported and still corrected.

Known deviation from server mode

When the manifests genuinely disagree about a field the API server defaults, client mode reports it as removed where server mode shows it changing to the default — a chart that stops pinning replicas: 3 shows the field going away rather than 31. The change is reported either way; client mode just cannot name the value that replaces it. Mutating webhooks and validation are also not applied. All of this is documented in the new README section, along with the minimal RBAC role.

Tests

manifest/generate_test.go is new. Eight TestLocalMerge_* cases are built from real manifests (unstructured, as kube.Client.Build supplies them; the live side written as the API server returns it) and cover each symptom above, plus:

  • a chart that genuinely stops setting a field — still reported as removed;
  • a hand-scaled replicas and hand-edited image in the cluster — still reset to the chart values, i.e. drift detection intact.

I checked the tests are not vacuous by reverting each fix independently; each fails without the step it covers. cmd/upgrade_test.go adds flag/env validation cases.

go test ./..., go vet and gofmt pass, and the README flag tables are regenerated with make readme.

Testing done

Built with make install/helm and run against a real cluster with an account lacking patch, on a release of ~15 resources (LibreChat and its subcharts). Before the two post-processing steps the client-side output differed from --three-way-merge-mode=server on defaulted fields; the cases in the tests are taken from that output.

🤖 Generated with Claude Code

oreonland others added 2 commits August 31, 2026 11:23
--three-way-merge computed the merge patch locally but asked the API
server to apply it as a dry-run, which needs the `patch` verb on every
diffed resource. A read-only account got:
cannot patch "x" with kind Deployment: ... is forbidden
The patch is now applied locally when the server round-trip is not
available, using the same strategic-merge (or JSON merge patch, for
custom resources) logic the API server would use. Only `get` is
required.
A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE)
selects between:
auto server dry-run, falling back to the local merge on Forbidden
or MethodNotAllowed. Any other error still aborts, so a
genuinely bad patch is not masked. The default.
server the previous behaviour; a missing permission is a hard error.
client never sends the patch at all.
Two post-processing steps keep the local result close to what the API
server returns, since client-go does not ship the defaulting functions:
* The merged object is round-tripped through its Go type, the way the
API server does before it answers. That drops the empty values a
manifest spells out but the type omits - `initialDelaySeconds: 0`,
`hostNetwork: false`, `sysctls: []` - which would otherwise show up
as additions the upgrade does not make.
* A field is copied back from the live object when the old and the new
release manifest agree about it. The patch replaces `retainKeys`
structs and atomic lists as a whole, and a manifest that renders an
unset value as an explicit `null` (a bare `replicas:`) reaches the
patch as a change rather than a deletion; both wipe values the API
server had defaulted in and re-defaults immediately after. Fields
the two manifests disagree about are left deleted, because that is a
change the chart really makes.
manifest.Generate takes variadic options rather than a new parameter, so
existing callers keep compiling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A chart that renders an empty collection literally - `rules: []`, the
branch grafana's Role and ClusterRole take when no sidecar is enabled -
disagrees with the cluster about how to write "nothing". Kubernetes
stores objects as protobuf, which cannot tell an empty repeated field
from an absent one, so the API server answers with `rules: null`.
The patch then carries `{"rules":[]}` even though the chart did not
change, and the client-side merge reported:
rules (rbac.authorization.k8s.io/v1/Role/grafana/grafana)
± type change from <nil> to list
server mode shows nothing, because its answer goes back through the same
storage round-trip. The local merge now keeps whichever spelling the
cluster reports whenever the merged and the live value are both empty.
Emptying a collection that actually had entries is still a change and is
still reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

auto mode does not currently “fall back per run” (it keeps attempting forbidden PATCHes) and the docs/RBAC wording plus env-var handling need small corrections to match intended behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new three-way-merge “mode” capability so helm diff upgrade --three-way-merge can work with read-only Kubernetes credentials by falling back (or opting in) to a client-side patch application that approximates the API server’s results.

Changes:

  • Introduces --three-way-merge-mode (auto/server/client) and env HELM_DIFF_THREE_WAY_MERGE_MODE, wiring it through cmd/upgrade into manifest.Generate(...) via variadic options.
  • Implements client-side patch application for three-way-merge, including normalization (round-trip through Go types) and a restoration pass for server-populated/defaulted fields.
  • Adds focused unit tests for local merge behavior and CLI flag/env validation; updates README and Copilot instructions.
File summaries
FileDescription
README.mdDocuments --three-way-merge-mode, behavior tradeoffs, and RBAC guidance.
manifest/generate.goAdds merge-mode options and client-side patch application with post-processing to reduce diff noise.
manifest/generate_test.goNew tests covering strategic/merge patch local application and default-restoration behaviors.
cmd/upgrade.goAdds flag/env plumbing and validation; passes merge-mode into manifest generation.
cmd/upgrade_test.goAdds validation coverage for the new flag/env behavior.
.github/copilot-instructions.mdDocuments the new environment variable.
Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadmanifest/generate.go Outdated
Comment threadcmd/upgrade.go Outdated
Comment threadREADME.md Outdated
Comment threadmanifest/generate.go Outdated
auto mode now falls back for the whole run, as documented. It warned
once but kept sending a dry-run PATCH for every remaining resource, so a
release of N objects produced N denied requests - N audit log entries and
N round-trips - to reach the same conclusion each time. A refused dry-run
says something about the credentials rather than about the resource, so
the first refusal now moves the rest of the run to the local merge. The
decision moved into a clientSideFallback type so it can be tested without
a cluster.
HELM_DIFF_THREE_WAY_MERGE_MODE is read only when the run actually
performs a three-way merge (--three-way-merge, or --take-ownership which
implies it). A stale value in the environment could otherwise fail an
unrelated `helm diff upgrade` over a mode it was never going to use. The
flag stays validated unconditionally: passing it is deliberate, so a typo
is worth reporting either way.
The README no longer calls a wildcard role "minimal RBAC" - it grants
read access to every resource in every API group. It is now described as
the blunt option, next to a scoped example and a note that a kind missing
from the role fails the diff for that resource.
Also fixes the grammar in the two createPatch comments the review
flagged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new validation error messages format the allowed-mode slice with %q, which will render as %!q([]string=...) and should be corrected before merge.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

cmd/upgrade.go:199

  • The format string uses %q for manifest.ValidThreeWayMergeModes (a []string), which will render as %!q([]string=...) and produce an unhelpful validation error. Use %v (or join the slice) when printing the allowed values.

This issue also appears on line 226 of the same file.

cmd/upgrade.go:228

  • The env-var validation error formats manifest.ValidThreeWayMergeModes (a []string) with %q, which will print as %!q([]string=...). Use %v (or join the slice) so users see the allowed values clearly.
 if !slices.Contains(manifest.ValidThreeWayMergeModes, mode) {
return fmt.Errorf("env var %q must be one of %q, but got %q", "HELM_DIFF_THREE_WAY_MERGE_MODE", manifest.ValidThreeWayMergeModes, mode)
}
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@oreonl

Copy link
Copy Markdown
Author

Checked this one and I don't think it holds — %q on a []string isn't malformed. fmt applies a verb recursively to the elements of a compound operand (docs: "For compound objects, the elements are printed using these rules, recursively... array, slice: [elem0 elem1 ...]"). %!q(...) only appears when the verb is invalid for the type, e.g. %q on a struct.

Actual output from the current branch, for both lines flagged:

$ helm diff upgrade rel ./chart --three-way-merge-mode local
Error: flag "three-way-merge-mode" must be one of ["auto" "server" "client"], but got "local"
$ HELM_DIFF_THREE_WAY_MERGE_MODE=local helm diff upgrade rel ./chart --three-way-merge
Error: env var "HELM_DIFF_THREE_WAY_MERGE_MODE" must be one of ["auto" "server" "client"], but got "local"

%v would print [auto server client], dropping the quotes that delimit the values — and the adjacent --server-side check quotes each one:

Error: flag "server-side" must be "true", "false" or "auto", but got "yes"

So I've left these as they are.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core diff computation behavior with non-trivial Kubernetes patch/defaulting semantics, so it warrants final human review despite good test coverage.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment threadmanifest/generate.go Outdated
The marshal of originalObj reported "serializing current configuration",
while the live object marshalled a few lines below reported "serializing
live configuration". Two different objects, and the first was named after
the second, which is misleading when a failure has to be diagnosed. The
three messages now say original, target and live.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core three-way-merge behavior with substantial new merge/restoration logic whose correctness and compatibility impact are difficult to fully validate via automated review alone.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@yxxhero

Copy link
Copy Markdown
Collaborator

@oreonl PTAL

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Local restoration can hide real drift and CRD changes, while automatic fallback can mask admission failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadREADME.md Outdated
oreonland others added 2 commits September 3, 2026 13:55
Four defects from review, all confirmed by reproduction before the fix.
restoreMissing copied a live value back whenever the two release
manifests agreed about a key, including when they agreed on an actual
value. normalize drops an explicit `hostNetwork: false` through
omitempty, so a chart pinning false against a cluster that drifted to
true had the drift restored and the correction vanished from the diff -
the opposite of what three-way merge is for. A value is now restored only
where neither manifest carries one, an absent key and a `null` counting
alike, which is all the unset `replicas:` case ever needed.
The restoration pass no longer runs behind a JSON merge patch. That patch
carries only what the chart changed between the two manifests, so it
never prunes a field the cluster populated and there is nothing to put
back. Custom resources are also stored as JSON rather than protobuf,
which keeps `null`, `[]` and `{}` apart, so the empty-value equivalence
was undoing a chart deliberately changing one into another.
auto no longer reads every 403 as a missing patch permission. An
admission webhook or a quota controller can refuse a request the
credentials are entitled to make, and that refusal is exactly what the
upgrade would hit, so falling back locally hid the outcome the diff
exists to predict. A SelfSubjectAccessReview now settles whether patching
is permitted before the refusal is treated as a permission problem; where
the review cannot be made the previous behaviour stands. The fallback is
already once per run, so this costs one extra request.
The README documents a second deviation from server mode: changing an
entry inside an atomic list also drops the server defaults elsewhere in
that entry, because positions in a list the chart rewrote cannot be
matched up safely.
k8s.io/api/authorization/v1 is added to the depguard allowlist, which
golangci-lint requires for the new import.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fallback handling and field restoration have correctness gaps, and several usage and RBAC examples are inaccurate.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (5)

Previously missed (3) — in code that hasn't changed since the last review.

README.md:276

  • The release-storage requirement is understated here: the default revision is resolved by helm get manifest without --revision, which lists the Helm storage backend to find the newest release. A role with only get on Secrets/ConfigMaps therefore fails for the normal invocation; document list as required as well.

This issue also appears on line 297 of the same file.
README.md:404

  • This generated example does not enable three-way merge. Since the mode environment variable is intentionally ignored unless three-way merge is already active, the command performs a normal two-way diff and does not demonstrate client-side merging.
    cmd/upgrade.go:170
  • This example does not enable three-way merge. The command only reads this environment variable when HELM_DIFF_THREE_WAY_MERGE=true (or --three-way-merge/--take-ownership) is already active, so as written it performs the ordinary two-way diff rather than a local merge.

manifest/generate.go:233

  • MethodNotAllowed never falls back when the authorization review says patching is allowed: isPatchNotAllowed routes it here, then this check returns false and aborts. That contradicts the new auto contract, which explicitly falls back for MethodNotAllowed. Restrict the authorization disambiguation to Forbidden; a 405 indicates the server path is unavailable regardless of RBAC.
	if f.canPatch != nil {
if allowed, err := f.canPatch(info); err == nil && allowed {
return false
}

README.md:299

  • The least-privilege example cannot read the latest Helm release with the default revision. This code invokes helm get manifest without --revision (cmd/helm.go:139-158), so Helm must list the active Secret/ConfigMap storage backend to find the newest revision; granting only get causes the diff to fail before resource reads. Grant list to release storage while keeping workload kinds at get.
 - apiGroups: [""]
resources: ["configmaps", "secrets", "services", "serviceaccounts"]
verbs: ["get"]
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment threadmanifest/generate.go
A MethodNotAllowed no longer aborts. The authorization review added in
the previous commit ran for every refusal isPatchNotAllowed accepts, so a
405 against credentials that may patch was read as "not a permission
problem" and surfaced. A 405 says the API server takes no patch for the
resource whoever is asking, so being entitled to send one changes
nothing; only a 403 is ambiguous enough to be worth the review.
The least-privilege role could not read the release. Without an explicit
--revision helm lists the storage backend to find the newest one, so
`get` on Secrets and ConfigMaps is not enough - the release storage needs
`list` as well, while the kinds a chart renders still only need `get`.
The HELM_DIFF_THREE_WAY_MERGE_MODE examples did not enable the three-way
merge, so they demonstrated an ordinary two-way diff. Now that the
variable is deliberately ignored unless the merge is on, they have to
pass --three-way-merge to show anything. The README copy of the example
block is maintained by hand rather than by scripts/gen-readme.sh, which
only regenerates the flag tables, so it had drifted from the binary.
The README also documents that client mode can be quieter than the truth
in one place: the restoration copies back every live-only field a
wholesale replacement removed, and nothing local can tell a value the API
server defaulted from one somebody set by hand. A NetworkPolicy port with
a hand-added endPort keeps it, where the upgrade would drop it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oreonl

Copy link
Copy Markdown
Author

All five points from this review are addressed in d8d73ff. The four suppressed ones have no threads, so answering them here.

MethodNotAllowed aborting instead of falling back (manifest/generate.go:233) — correct, and a regression from the SelfSubjectAccessReview I added in ca059ad: the review ran for every refusal isPatchNotAllowed accepts, so a 405 against credentials that may patch was surfaced rather than falling back. Reproduced (patchDenied(405, canPatch=true) returned false) and fixed by restricting the disambiguation to Forbidden — a 405 says the API server takes no patch for that resource whoever is asking, so entitlement changes nothing. Covered by TestClientSideFallbackAlwaysFallsBackOnMethodNotAllowed.

Release storage needs list (README.md:276, :297, :299) — confirmed in the code: helmGetArgs omits --revision when the revision is 0 (cmd/helm.go:141-145), and helm's Secrets driver Query does a List to find the newest revision, so get alone fails before any resource is read. The least-privilege example now grants get, list on configmaps/secrets for the release storage and keeps the rendered kinds at get, with the reason spelled out in the surrounding prose.

The env var examples did not enable the three-way merge (README.md:404, cmd/upgrade.go:170) — right, and self-inflicted: making HELM_DIFF_THREE_WAY_MERGE_MODE deliberately ignored unless the merge is on invalidated my own examples. Both now pass --three-way-merge. Worth flagging for future edits that scripts/gen-readme.sh regenerates only the flag tables, so the README's copy of the example block is hand-maintained and had drifted from the binary; I diffed the two to confirm they match again.

The remaining inline comment, about live-only drift inside a replaced atomic list, is answered on its own thread. I have left that one open, since it is a design trade-off rather than something I have fixed.

For what it is worth while the checks are still pending: golangci-lint v2.12.2 against Go 1.26.4 — the pairing lint.yaml pins via go-version-file — reports 0 issues, and go test ./... passes.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Local normalization can discard newer Kubernetes fields, while explicit-null restoration can hide real drift corrections.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

manifest/generate.go:365

  • Decoding into the compiled Go type silently discards fields unknown to this client-go version. If helm-diff runs against a newer cluster and a chart adds a newly supported field to an otherwise known built-in kind, the local merge contains the field here, this round-trip removes it, and the restoration pass cannot recover it because it is absent from the live object. Client mode then omits a real upgrade change; preserve unknown fields from the merged object during normalization or avoid typed normalization when the typed schema does not cover the object.

manifest/generate.go:474

  • This treats an explicit null exactly like a missing key, but null can be the chart-owned desired value for fields that are not defaulted. For example, if both manifests render runtimeClassName: null while the live Pod template drifted to runtimeClassName: gvisor, the three-way patch clears the field and normalize omits it; this condition then restores gvisor, hiding a correction that the real upgrade makes. Use presence-aware lookups and restrict null restoration to fields known to be server-defaulted rather than restoring every explicit-null field.
 if originalMap[key] == nil && modifiedMap[key] == nil {
mergedMap[key] = liveValue
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@oreonl@yxxhero
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat: make --three-way-merge work without patch permissions - #1063

Open
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission
Open

feat: make --three-way-merge work without patch permissions#1063
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission

Conversation

@oreonl

Copy link
Copy Markdown

What

--three-way-merge currently requires the patch permission on every diffed resource. It computes the merge patch locally but asks the API server to apply it as a dry-run:

helper.ServerDryRun=truetargetObj, err:=helper.Patch(info.Namespace, info.Name, patchType, patch, nil)

A read-only account gets cannot patch "x" with kind Deployment: ... is forbidden, so the feature is unavailable to exactly the credentials a "show me what this upgrade would do" tool tends to run under.

This applies the patch locally when the server round-trip is not available, using the same strategic-merge (or JSON merge patch, for custom resources) logic the API server would use. Only get is required.

Interface

A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE):

modebehaviour
auto (default)server dry-run, falling back to the local merge on Forbidden / MethodNotAllowed. Any other error still aborts, so a genuinely bad patch is not masked.
serverthe previous behaviour — a missing patch permission is a hard error.
clientnever sends the patch at all.

auto means a read-only account works out of the box; it only degrades when the accurate path is unavailable anyway, and says so on stderr.

manifest.Generate takes variadic options rather than a new parameter, so existing callers (helmfile vendors this package) keep compiling.

Matching what the server returns

A naive local merge produces a diff full of noise, because the API server does more than apply the patch. scheme.Scheme.Default() is a no-op in client-go — the SetDefaults_* functions live in k8s.io/kubernetes and are not importable — so the defaults cannot simply be recomputed. Two post-processing steps recover most of the gap:

1. Round-trip through the Go type, the way the API server does before it answers. That drops the empty values a manifest spells out but the type omits — initialDelaySeconds: 0, hostNetwork: false, sysctls: [], supplementalGroups: [] — which otherwise appear as additions the upgrade does not make. (Helm's kube.Client.Build returns unstructured objects, so these survive from the chart YAML verbatim.)

2. Copy a field back from the live object when the old and the new release manifest agree about it. Values disappear from the merged object two ways:

  • the patch replaces retainKeys structs (spec.strategy) and atomic lists (ports, volumeClaimTemplates) as a whole, dropping what the server had defaulted into them;
  • a manifest that renders an unset value as an explicit null — a bare replicas:, which is what a chart writes for a value it leaves unset — puts the key in both manifests, so it reaches the patch as a change rather than a deletion. That deletes a defaulted value even when the chart did not change at all.

The server re-defaults both cases immediately after applying the patch. Locally, a field is restored only when nothing in either manifest asked for it to go. Where the two manifests disagree the deletion is honoured, because that is a change the chart really makes — restoring there would hide real removals (a dropped nodeSelector entry looks identical at the JSON level, and the server does not default that back). Values already present in the merged object are never overwritten, so drift between the cluster and the chart is still reported and still corrected.

Known deviation from server mode

When the manifests genuinely disagree about a field the API server defaults, client mode reports it as removed where server mode shows it changing to the default — a chart that stops pinning replicas: 3 shows the field going away rather than 31. The change is reported either way; client mode just cannot name the value that replaces it. Mutating webhooks and validation are also not applied. All of this is documented in the new README section, along with the minimal RBAC role.

Tests

manifest/generate_test.go is new. Eight TestLocalMerge_* cases are built from real manifests (unstructured, as kube.Client.Build supplies them; the live side written as the API server returns it) and cover each symptom above, plus:

  • a chart that genuinely stops setting a field — still reported as removed;
  • a hand-scaled replicas and hand-edited image in the cluster — still reset to the chart values, i.e. drift detection intact.

I checked the tests are not vacuous by reverting each fix independently; each fails without the step it covers. cmd/upgrade_test.go adds flag/env validation cases.

go test ./..., go vet and gofmt pass, and the README flag tables are regenerated with make readme.

Testing done

Built with make install/helm and run against a real cluster with an account lacking patch, on a release of ~15 resources (LibreChat and its subcharts). Before the two post-processing steps the client-side output differed from --three-way-merge-mode=server on defaulted fields; the cases in the tests are taken from that output.

🤖 Generated with Claude Code

oreonland others added 2 commits August 31, 2026 11:23
--three-way-merge computed the merge patch locally but asked the API
server to apply it as a dry-run, which needs the `patch` verb on every
diffed resource. A read-only account got:
cannot patch "x" with kind Deployment: ... is forbidden
The patch is now applied locally when the server round-trip is not
available, using the same strategic-merge (or JSON merge patch, for
custom resources) logic the API server would use. Only `get` is
required.
A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE)
selects between:
auto server dry-run, falling back to the local merge on Forbidden
or MethodNotAllowed. Any other error still aborts, so a
genuinely bad patch is not masked. The default.
server the previous behaviour; a missing permission is a hard error.
client never sends the patch at all.
Two post-processing steps keep the local result close to what the API
server returns, since client-go does not ship the defaulting functions:
* The merged object is round-tripped through its Go type, the way the
API server does before it answers. That drops the empty values a
manifest spells out but the type omits - `initialDelaySeconds: 0`,
`hostNetwork: false`, `sysctls: []` - which would otherwise show up
as additions the upgrade does not make.
* A field is copied back from the live object when the old and the new
release manifest agree about it. The patch replaces `retainKeys`
structs and atomic lists as a whole, and a manifest that renders an
unset value as an explicit `null` (a bare `replicas:`) reaches the
patch as a change rather than a deletion; both wipe values the API
server had defaulted in and re-defaults immediately after. Fields
the two manifests disagree about are left deleted, because that is a
change the chart really makes.
manifest.Generate takes variadic options rather than a new parameter, so
existing callers keep compiling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A chart that renders an empty collection literally - `rules: []`, the
branch grafana's Role and ClusterRole take when no sidecar is enabled -
disagrees with the cluster about how to write "nothing". Kubernetes
stores objects as protobuf, which cannot tell an empty repeated field
from an absent one, so the API server answers with `rules: null`.
The patch then carries `{"rules":[]}` even though the chart did not
change, and the client-side merge reported:
rules (rbac.authorization.k8s.io/v1/Role/grafana/grafana)
± type change from <nil> to list
server mode shows nothing, because its answer goes back through the same
storage round-trip. The local merge now keeps whichever spelling the
cluster reports whenever the merged and the live value are both empty.
Emptying a collection that actually had entries is still a change and is
still reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

auto mode does not currently “fall back per run” (it keeps attempting forbidden PATCHes) and the docs/RBAC wording plus env-var handling need small corrections to match intended behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new three-way-merge “mode” capability so helm diff upgrade --three-way-merge can work with read-only Kubernetes credentials by falling back (or opting in) to a client-side patch application that approximates the API server’s results.

Changes:

  • Introduces --three-way-merge-mode (auto/server/client) and env HELM_DIFF_THREE_WAY_MERGE_MODE, wiring it through cmd/upgrade into manifest.Generate(...) via variadic options.
  • Implements client-side patch application for three-way-merge, including normalization (round-trip through Go types) and a restoration pass for server-populated/defaulted fields.
  • Adds focused unit tests for local merge behavior and CLI flag/env validation; updates README and Copilot instructions.
File summaries
FileDescription
README.mdDocuments --three-way-merge-mode, behavior tradeoffs, and RBAC guidance.
manifest/generate.goAdds merge-mode options and client-side patch application with post-processing to reduce diff noise.
manifest/generate_test.goNew tests covering strategic/merge patch local application and default-restoration behaviors.
cmd/upgrade.goAdds flag/env plumbing and validation; passes merge-mode into manifest generation.
cmd/upgrade_test.goAdds validation coverage for the new flag/env behavior.
.github/copilot-instructions.mdDocuments the new environment variable.
Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadmanifest/generate.go Outdated
Comment threadcmd/upgrade.go Outdated
Comment threadREADME.md Outdated
Comment threadmanifest/generate.go Outdated
auto mode now falls back for the whole run, as documented. It warned
once but kept sending a dry-run PATCH for every remaining resource, so a
release of N objects produced N denied requests - N audit log entries and
N round-trips - to reach the same conclusion each time. A refused dry-run
says something about the credentials rather than about the resource, so
the first refusal now moves the rest of the run to the local merge. The
decision moved into a clientSideFallback type so it can be tested without
a cluster.
HELM_DIFF_THREE_WAY_MERGE_MODE is read only when the run actually
performs a three-way merge (--three-way-merge, or --take-ownership which
implies it). A stale value in the environment could otherwise fail an
unrelated `helm diff upgrade` over a mode it was never going to use. The
flag stays validated unconditionally: passing it is deliberate, so a typo
is worth reporting either way.
The README no longer calls a wildcard role "minimal RBAC" - it grants
read access to every resource in every API group. It is now described as
the blunt option, next to a scoped example and a note that a kind missing
from the role fails the diff for that resource.
Also fixes the grammar in the two createPatch comments the review
flagged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new validation error messages format the allowed-mode slice with %q, which will render as %!q([]string=...) and should be corrected before merge.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

cmd/upgrade.go:199

  • The format string uses %q for manifest.ValidThreeWayMergeModes (a []string), which will render as %!q([]string=...) and produce an unhelpful validation error. Use %v (or join the slice) when printing the allowed values.

This issue also appears on line 226 of the same file.

cmd/upgrade.go:228

  • The env-var validation error formats manifest.ValidThreeWayMergeModes (a []string) with %q, which will print as %!q([]string=...). Use %v (or join the slice) so users see the allowed values clearly.
 if !slices.Contains(manifest.ValidThreeWayMergeModes, mode) {
return fmt.Errorf("env var %q must be one of %q, but got %q", "HELM_DIFF_THREE_WAY_MERGE_MODE", manifest.ValidThreeWayMergeModes, mode)
}
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@oreonl

Copy link
Copy Markdown
Author

Checked this one and I don't think it holds — %q on a []string isn't malformed. fmt applies a verb recursively to the elements of a compound operand (docs: "For compound objects, the elements are printed using these rules, recursively... array, slice: [elem0 elem1 ...]"). %!q(...) only appears when the verb is invalid for the type, e.g. %q on a struct.

Actual output from the current branch, for both lines flagged:

$ helm diff upgrade rel ./chart --three-way-merge-mode local
Error: flag "three-way-merge-mode" must be one of ["auto" "server" "client"], but got "local"
$ HELM_DIFF_THREE_WAY_MERGE_MODE=local helm diff upgrade rel ./chart --three-way-merge
Error: env var "HELM_DIFF_THREE_WAY_MERGE_MODE" must be one of ["auto" "server" "client"], but got "local"

%v would print [auto server client], dropping the quotes that delimit the values — and the adjacent --server-side check quotes each one:

Error: flag "server-side" must be "true", "false" or "auto", but got "yes"

So I've left these as they are.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core diff computation behavior with non-trivial Kubernetes patch/defaulting semantics, so it warrants final human review despite good test coverage.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment threadmanifest/generate.go Outdated
The marshal of originalObj reported "serializing current configuration",
while the live object marshalled a few lines below reported "serializing
live configuration". Two different objects, and the first was named after
the second, which is misleading when a failure has to be diagnosed. The
three messages now say original, target and live.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core three-way-merge behavior with substantial new merge/restoration logic whose correctness and compatibility impact are difficult to fully validate via automated review alone.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@yxxhero

Copy link
Copy Markdown
Collaborator

@oreonl PTAL

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Local restoration can hide real drift and CRD changes, while automatic fallback can mask admission failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadREADME.md Outdated
oreonland others added 2 commits September 3, 2026 13:55
Four defects from review, all confirmed by reproduction before the fix.
restoreMissing copied a live value back whenever the two release
manifests agreed about a key, including when they agreed on an actual
value. normalize drops an explicit `hostNetwork: false` through
omitempty, so a chart pinning false against a cluster that drifted to
true had the drift restored and the correction vanished from the diff -
the opposite of what three-way merge is for. A value is now restored only
where neither manifest carries one, an absent key and a `null` counting
alike, which is all the unset `replicas:` case ever needed.
The restoration pass no longer runs behind a JSON merge patch. That patch
carries only what the chart changed between the two manifests, so it
never prunes a field the cluster populated and there is nothing to put
back. Custom resources are also stored as JSON rather than protobuf,
which keeps `null`, `[]` and `{}` apart, so the empty-value equivalence
was undoing a chart deliberately changing one into another.
auto no longer reads every 403 as a missing patch permission. An
admission webhook or a quota controller can refuse a request the
credentials are entitled to make, and that refusal is exactly what the
upgrade would hit, so falling back locally hid the outcome the diff
exists to predict. A SelfSubjectAccessReview now settles whether patching
is permitted before the refusal is treated as a permission problem; where
the review cannot be made the previous behaviour stands. The fallback is
already once per run, so this costs one extra request.
The README documents a second deviation from server mode: changing an
entry inside an atomic list also drops the server defaults elsewhere in
that entry, because positions in a list the chart rewrote cannot be
matched up safely.
k8s.io/api/authorization/v1 is added to the depguard allowlist, which
golangci-lint requires for the new import.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fallback handling and field restoration have correctness gaps, and several usage and RBAC examples are inaccurate.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (5)

Previously missed (3) — in code that hasn't changed since the last review.

README.md:276

  • The release-storage requirement is understated here: the default revision is resolved by helm get manifest without --revision, which lists the Helm storage backend to find the newest release. A role with only get on Secrets/ConfigMaps therefore fails for the normal invocation; document list as required as well.

This issue also appears on line 297 of the same file.
README.md:404

  • This generated example does not enable three-way merge. Since the mode environment variable is intentionally ignored unless three-way merge is already active, the command performs a normal two-way diff and does not demonstrate client-side merging.
    cmd/upgrade.go:170
  • This example does not enable three-way merge. The command only reads this environment variable when HELM_DIFF_THREE_WAY_MERGE=true (or --three-way-merge/--take-ownership) is already active, so as written it performs the ordinary two-way diff rather than a local merge.

manifest/generate.go:233

  • MethodNotAllowed never falls back when the authorization review says patching is allowed: isPatchNotAllowed routes it here, then this check returns false and aborts. That contradicts the new auto contract, which explicitly falls back for MethodNotAllowed. Restrict the authorization disambiguation to Forbidden; a 405 indicates the server path is unavailable regardless of RBAC.
	if f.canPatch != nil {
if allowed, err := f.canPatch(info); err == nil && allowed {
return false
}

README.md:299

  • The least-privilege example cannot read the latest Helm release with the default revision. This code invokes helm get manifest without --revision (cmd/helm.go:139-158), so Helm must list the active Secret/ConfigMap storage backend to find the newest revision; granting only get causes the diff to fail before resource reads. Grant list to release storage while keeping workload kinds at get.
 - apiGroups: [""]
resources: ["configmaps", "secrets", "services", "serviceaccounts"]
verbs: ["get"]
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment threadmanifest/generate.go
A MethodNotAllowed no longer aborts. The authorization review added in
the previous commit ran for every refusal isPatchNotAllowed accepts, so a
405 against credentials that may patch was read as "not a permission
problem" and surfaced. A 405 says the API server takes no patch for the
resource whoever is asking, so being entitled to send one changes
nothing; only a 403 is ambiguous enough to be worth the review.
The least-privilege role could not read the release. Without an explicit
--revision helm lists the storage backend to find the newest one, so
`get` on Secrets and ConfigMaps is not enough - the release storage needs
`list` as well, while the kinds a chart renders still only need `get`.
The HELM_DIFF_THREE_WAY_MERGE_MODE examples did not enable the three-way
merge, so they demonstrated an ordinary two-way diff. Now that the
variable is deliberately ignored unless the merge is on, they have to
pass --three-way-merge to show anything. The README copy of the example
block is maintained by hand rather than by scripts/gen-readme.sh, which
only regenerates the flag tables, so it had drifted from the binary.
The README also documents that client mode can be quieter than the truth
in one place: the restoration copies back every live-only field a
wholesale replacement removed, and nothing local can tell a value the API
server defaulted from one somebody set by hand. A NetworkPolicy port with
a hand-added endPort keeps it, where the upgrade would drop it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oreonl

Copy link
Copy Markdown
Author

All five points from this review are addressed in d8d73ff. The four suppressed ones have no threads, so answering them here.

MethodNotAllowed aborting instead of falling back (manifest/generate.go:233) — correct, and a regression from the SelfSubjectAccessReview I added in ca059ad: the review ran for every refusal isPatchNotAllowed accepts, so a 405 against credentials that may patch was surfaced rather than falling back. Reproduced (patchDenied(405, canPatch=true) returned false) and fixed by restricting the disambiguation to Forbidden — a 405 says the API server takes no patch for that resource whoever is asking, so entitlement changes nothing. Covered by TestClientSideFallbackAlwaysFallsBackOnMethodNotAllowed.

Release storage needs list (README.md:276, :297, :299) — confirmed in the code: helmGetArgs omits --revision when the revision is 0 (cmd/helm.go:141-145), and helm's Secrets driver Query does a List to find the newest revision, so get alone fails before any resource is read. The least-privilege example now grants get, list on configmaps/secrets for the release storage and keeps the rendered kinds at get, with the reason spelled out in the surrounding prose.

The env var examples did not enable the three-way merge (README.md:404, cmd/upgrade.go:170) — right, and self-inflicted: making HELM_DIFF_THREE_WAY_MERGE_MODE deliberately ignored unless the merge is on invalidated my own examples. Both now pass --three-way-merge. Worth flagging for future edits that scripts/gen-readme.sh regenerates only the flag tables, so the README's copy of the example block is hand-maintained and had drifted from the binary; I diffed the two to confirm they match again.

The remaining inline comment, about live-only drift inside a replaced atomic list, is answered on its own thread. I have left that one open, since it is a design trade-off rather than something I have fixed.

For what it is worth while the checks are still pending: golangci-lint v2.12.2 against Go 1.26.4 — the pairing lint.yaml pins via go-version-file — reports 0 issues, and go test ./... passes.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Local normalization can discard newer Kubernetes fields, while explicit-null restoration can hide real drift corrections.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

manifest/generate.go:365

  • Decoding into the compiled Go type silently discards fields unknown to this client-go version. If helm-diff runs against a newer cluster and a chart adds a newly supported field to an otherwise known built-in kind, the local merge contains the field here, this round-trip removes it, and the restoration pass cannot recover it because it is absent from the live object. Client mode then omits a real upgrade change; preserve unknown fields from the merged object during normalization or avoid typed normalization when the typed schema does not cover the object.

manifest/generate.go:474

  • This treats an explicit null exactly like a missing key, but null can be the chart-owned desired value for fields that are not defaulted. For example, if both manifests render runtimeClassName: null while the live Pod template drifted to runtimeClassName: gvisor, the three-way patch clears the field and normalize omits it; this condition then restores gvisor, hiding a correction that the real upgrade makes. Use presence-aware lookups and restrict null restoration to fields known to be server-defaulted rather than restoring every explicit-null field.
 if originalMap[key] == nil && modifiedMap[key] == nil {
mergedMap[key] = liveValue
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@oreonl@yxxhero
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

feat: make --three-way-merge work without patch permissions - #1063

Open
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission
Open

feat: make --three-way-merge work without patch permissions#1063
oreonl wants to merge 7 commits into
databus23:masterfrom
oreonl:three-way-merge-without-patch-permission

Conversation

@oreonl

Copy link
Copy Markdown

What

--three-way-merge currently requires the patch permission on every diffed resource. It computes the merge patch locally but asks the API server to apply it as a dry-run:

helper.ServerDryRun=truetargetObj, err:=helper.Patch(info.Namespace, info.Name, patchType, patch, nil)

A read-only account gets cannot patch "x" with kind Deployment: ... is forbidden, so the feature is unavailable to exactly the credentials a "show me what this upgrade would do" tool tends to run under.

This applies the patch locally when the server round-trip is not available, using the same strategic-merge (or JSON merge patch, for custom resources) logic the API server would use. Only get is required.

Interface

A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE):

modebehaviour
auto (default)server dry-run, falling back to the local merge on Forbidden / MethodNotAllowed. Any other error still aborts, so a genuinely bad patch is not masked.
serverthe previous behaviour — a missing patch permission is a hard error.
clientnever sends the patch at all.

auto means a read-only account works out of the box; it only degrades when the accurate path is unavailable anyway, and says so on stderr.

manifest.Generate takes variadic options rather than a new parameter, so existing callers (helmfile vendors this package) keep compiling.

Matching what the server returns

A naive local merge produces a diff full of noise, because the API server does more than apply the patch. scheme.Scheme.Default() is a no-op in client-go — the SetDefaults_* functions live in k8s.io/kubernetes and are not importable — so the defaults cannot simply be recomputed. Two post-processing steps recover most of the gap:

1. Round-trip through the Go type, the way the API server does before it answers. That drops the empty values a manifest spells out but the type omits — initialDelaySeconds: 0, hostNetwork: false, sysctls: [], supplementalGroups: [] — which otherwise appear as additions the upgrade does not make. (Helm's kube.Client.Build returns unstructured objects, so these survive from the chart YAML verbatim.)

2. Copy a field back from the live object when the old and the new release manifest agree about it. Values disappear from the merged object two ways:

  • the patch replaces retainKeys structs (spec.strategy) and atomic lists (ports, volumeClaimTemplates) as a whole, dropping what the server had defaulted into them;
  • a manifest that renders an unset value as an explicit null — a bare replicas:, which is what a chart writes for a value it leaves unset — puts the key in both manifests, so it reaches the patch as a change rather than a deletion. That deletes a defaulted value even when the chart did not change at all.

The server re-defaults both cases immediately after applying the patch. Locally, a field is restored only when nothing in either manifest asked for it to go. Where the two manifests disagree the deletion is honoured, because that is a change the chart really makes — restoring there would hide real removals (a dropped nodeSelector entry looks identical at the JSON level, and the server does not default that back). Values already present in the merged object are never overwritten, so drift between the cluster and the chart is still reported and still corrected.

Known deviation from server mode

When the manifests genuinely disagree about a field the API server defaults, client mode reports it as removed where server mode shows it changing to the default — a chart that stops pinning replicas: 3 shows the field going away rather than 31. The change is reported either way; client mode just cannot name the value that replaces it. Mutating webhooks and validation are also not applied. All of this is documented in the new README section, along with the minimal RBAC role.

Tests

manifest/generate_test.go is new. Eight TestLocalMerge_* cases are built from real manifests (unstructured, as kube.Client.Build supplies them; the live side written as the API server returns it) and cover each symptom above, plus:

  • a chart that genuinely stops setting a field — still reported as removed;
  • a hand-scaled replicas and hand-edited image in the cluster — still reset to the chart values, i.e. drift detection intact.

I checked the tests are not vacuous by reverting each fix independently; each fails without the step it covers. cmd/upgrade_test.go adds flag/env validation cases.

go test ./..., go vet and gofmt pass, and the README flag tables are regenerated with make readme.

Testing done

Built with make install/helm and run against a real cluster with an account lacking patch, on a release of ~15 resources (LibreChat and its subcharts). Before the two post-processing steps the client-side output differed from --three-way-merge-mode=server on defaulted fields; the cases in the tests are taken from that output.

🤖 Generated with Claude Code

oreonland others added 2 commits August 31, 2026 11:23
--three-way-merge computed the merge patch locally but asked the API
server to apply it as a dry-run, which needs the `patch` verb on every
diffed resource. A read-only account got:
cannot patch "x" with kind Deployment: ... is forbidden
The patch is now applied locally when the server round-trip is not
available, using the same strategic-merge (or JSON merge patch, for
custom resources) logic the API server would use. Only `get` is
required.
A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE)
selects between:
auto server dry-run, falling back to the local merge on Forbidden
or MethodNotAllowed. Any other error still aborts, so a
genuinely bad patch is not masked. The default.
server the previous behaviour; a missing permission is a hard error.
client never sends the patch at all.
Two post-processing steps keep the local result close to what the API
server returns, since client-go does not ship the defaulting functions:
* The merged object is round-tripped through its Go type, the way the
API server does before it answers. That drops the empty values a
manifest spells out but the type omits - `initialDelaySeconds: 0`,
`hostNetwork: false`, `sysctls: []` - which would otherwise show up
as additions the upgrade does not make.
* A field is copied back from the live object when the old and the new
release manifest agree about it. The patch replaces `retainKeys`
structs and atomic lists as a whole, and a manifest that renders an
unset value as an explicit `null` (a bare `replicas:`) reaches the
patch as a change rather than a deletion; both wipe values the API
server had defaulted in and re-defaults immediately after. Fields
the two manifests disagree about are left deleted, because that is a
change the chart really makes.
manifest.Generate takes variadic options rather than a new parameter, so
existing callers keep compiling.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A chart that renders an empty collection literally - `rules: []`, the
branch grafana's Role and ClusterRole take when no sidecar is enabled -
disagrees with the cluster about how to write "nothing". Kubernetes
stores objects as protobuf, which cannot tell an empty repeated field
from an absent one, so the API server answers with `rules: null`.
The patch then carries `{"rules":[]}` even though the chart did not
change, and the client-side merge reported:
rules (rbac.authorization.k8s.io/v1/Role/grafana/grafana)
± type change from <nil> to list
server mode shows nothing, because its answer goes back through the same
storage round-trip. The local merge now keeps whichever spelling the
cluster reports whenever the merged and the live value are both empty.
Emptying a collection that actually had entries is still a change and is
still reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

auto mode does not currently “fall back per run” (it keeps attempting forbidden PATCHes) and the docs/RBAC wording plus env-var handling need small corrections to match intended behavior.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new three-way-merge “mode” capability so helm diff upgrade --three-way-merge can work with read-only Kubernetes credentials by falling back (or opting in) to a client-side patch application that approximates the API server’s results.

Changes:

  • Introduces --three-way-merge-mode (auto/server/client) and env HELM_DIFF_THREE_WAY_MERGE_MODE, wiring it through cmd/upgrade into manifest.Generate(...) via variadic options.
  • Implements client-side patch application for three-way-merge, including normalization (round-trip through Go types) and a restoration pass for server-populated/defaulted fields.
  • Adds focused unit tests for local merge behavior and CLI flag/env validation; updates README and Copilot instructions.
File summaries
FileDescription
README.mdDocuments --three-way-merge-mode, behavior tradeoffs, and RBAC guidance.
manifest/generate.goAdds merge-mode options and client-side patch application with post-processing to reduce diff noise.
manifest/generate_test.goNew tests covering strategic/merge patch local application and default-restoration behaviors.
cmd/upgrade.goAdds flag/env plumbing and validation; passes merge-mode into manifest generation.
cmd/upgrade_test.goAdds validation coverage for the new flag/env behavior.
.github/copilot-instructions.mdDocuments the new environment variable.
Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment threadmanifest/generate.go Outdated
Comment threadcmd/upgrade.go Outdated
Comment threadREADME.md Outdated
Comment threadmanifest/generate.go Outdated
auto mode now falls back for the whole run, as documented. It warned
once but kept sending a dry-run PATCH for every remaining resource, so a
release of N objects produced N denied requests - N audit log entries and
N round-trips - to reach the same conclusion each time. A refused dry-run
says something about the credentials rather than about the resource, so
the first refusal now moves the rest of the run to the local merge. The
decision moved into a clientSideFallback type so it can be tested without
a cluster.
HELM_DIFF_THREE_WAY_MERGE_MODE is read only when the run actually
performs a three-way merge (--three-way-merge, or --take-ownership which
implies it). A stale value in the environment could otherwise fail an
unrelated `helm diff upgrade` over a mode it was never going to use. The
flag stays validated unconditionally: passing it is deliberate, so a typo
is worth reporting either way.
The README no longer calls a wildcard role "minimal RBAC" - it grants
read access to every resource in every API group. It is now described as
the blunt option, next to a scoped example and a note that a kind missing
from the role fails the diff for that resource.
Also fixes the grammar in the two createPatch comments the review
flagged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new validation error messages format the allowed-mode slice with %q, which will render as %!q([]string=...) and should be corrected before merge.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

cmd/upgrade.go:199

  • The format string uses %q for manifest.ValidThreeWayMergeModes (a []string), which will render as %!q([]string=...) and produce an unhelpful validation error. Use %v (or join the slice) when printing the allowed values.

This issue also appears on line 226 of the same file.

cmd/upgrade.go:228

  • The env-var validation error formats manifest.ValidThreeWayMergeModes (a []string) with %q, which will print as %!q([]string=...). Use %v (or join the slice) so users see the allowed values clearly.
 if !slices.Contains(manifest.ValidThreeWayMergeModes, mode) {
return fmt.Errorf("env var %q must be one of %q, but got %q", "HELM_DIFF_THREE_WAY_MERGE_MODE", manifest.ValidThreeWayMergeModes, mode)
}
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@oreonl

Copy link
Copy Markdown
Author

Checked this one and I don't think it holds — %q on a []string isn't malformed. fmt applies a verb recursively to the elements of a compound operand (docs: "For compound objects, the elements are printed using these rules, recursively... array, slice: [elem0 elem1 ...]"). %!q(...) only appears when the verb is invalid for the type, e.g. %q on a struct.

Actual output from the current branch, for both lines flagged:

$ helm diff upgrade rel ./chart --three-way-merge-mode local
Error: flag "three-way-merge-mode" must be one of ["auto" "server" "client"], but got "local"
$ HELM_DIFF_THREE_WAY_MERGE_MODE=local helm diff upgrade rel ./chart --three-way-merge
Error: env var "HELM_DIFF_THREE_WAY_MERGE_MODE" must be one of ["auto" "server" "client"], but got "local"

%v would print [auto server client], dropping the quotes that delimit the values — and the adjacent --server-side check quotes each one:

Error: flag "server-side" must be "true", "false" or "auto", but got "yes"

So I've left these as they are.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core diff computation behavior with non-trivial Kubernetes patch/defaulting semantics, so it warrants final human review despite good test coverage.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment threadmanifest/generate.go Outdated
The marshal of originalObj reported "serializing current configuration",
while the live object marshalled a few lines below reported "serializing
live configuration". Two different objects, and the first was named after
the second, which is misleading when a failure has to be diagnosed. The
three messages now say original, target and live.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core three-way-merge behavior with substantial new merge/restoration logic whose correctness and compatibility impact are difficult to fully validate via automated review alone.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@yxxhero

Copy link
Copy Markdown
Collaborator

@oreonl PTAL

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Local restoration can hide real drift and CRD changes, while automatic fallback can mask admission failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadmanifest/generate.go
Comment threadREADME.md Outdated
oreonland others added 2 commits September 3, 2026 13:55
Four defects from review, all confirmed by reproduction before the fix.
restoreMissing copied a live value back whenever the two release
manifests agreed about a key, including when they agreed on an actual
value. normalize drops an explicit `hostNetwork: false` through
omitempty, so a chart pinning false against a cluster that drifted to
true had the drift restored and the correction vanished from the diff -
the opposite of what three-way merge is for. A value is now restored only
where neither manifest carries one, an absent key and a `null` counting
alike, which is all the unset `replicas:` case ever needed.
The restoration pass no longer runs behind a JSON merge patch. That patch
carries only what the chart changed between the two manifests, so it
never prunes a field the cluster populated and there is nothing to put
back. Custom resources are also stored as JSON rather than protobuf,
which keeps `null`, `[]` and `{}` apart, so the empty-value equivalence
was undoing a chart deliberately changing one into another.
auto no longer reads every 403 as a missing patch permission. An
admission webhook or a quota controller can refuse a request the
credentials are entitled to make, and that refusal is exactly what the
upgrade would hit, so falling back locally hid the outcome the diff
exists to predict. A SelfSubjectAccessReview now settles whether patching
is permitted before the refusal is treated as a permission problem; where
the review cannot be made the previous behaviour stands. The fallback is
already once per run, so this costs one extra request.
The README documents a second deviation from server mode: changing an
entry inside an atomic list also drops the server defaults elsewhere in
that entry, because positions in a list the chart rewrote cannot be
matched up safely.
k8s.io/api/authorization/v1 is added to the depguard allowlist, which
golangci-lint requires for the new import.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fallback handling and field restoration have correctness gaps, and several usage and RBAC examples are inaccurate.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (5)

Previously missed (3) — in code that hasn't changed since the last review.

README.md:276

  • The release-storage requirement is understated here: the default revision is resolved by helm get manifest without --revision, which lists the Helm storage backend to find the newest release. A role with only get on Secrets/ConfigMaps therefore fails for the normal invocation; document list as required as well.

This issue also appears on line 297 of the same file.
README.md:404

  • This generated example does not enable three-way merge. Since the mode environment variable is intentionally ignored unless three-way merge is already active, the command performs a normal two-way diff and does not demonstrate client-side merging.
    cmd/upgrade.go:170
  • This example does not enable three-way merge. The command only reads this environment variable when HELM_DIFF_THREE_WAY_MERGE=true (or --three-way-merge/--take-ownership) is already active, so as written it performs the ordinary two-way diff rather than a local merge.

manifest/generate.go:233

  • MethodNotAllowed never falls back when the authorization review says patching is allowed: isPatchNotAllowed routes it here, then this check returns false and aborts. That contradicts the new auto contract, which explicitly falls back for MethodNotAllowed. Restrict the authorization disambiguation to Forbidden; a 405 indicates the server path is unavailable regardless of RBAC.
	if f.canPatch != nil {
if allowed, err := f.canPatch(info); err == nil && allowed {
return false
}

README.md:299

  • The least-privilege example cannot read the latest Helm release with the default revision. This code invokes helm get manifest without --revision (cmd/helm.go:139-158), so Helm must list the active Secret/ConfigMap storage backend to find the newest revision; granting only get causes the diff to fail before resource reads. Grant list to release storage while keeping workload kinds at get.
 - apiGroups: [""]
resources: ["configmaps", "secrets", "services", "serviceaccounts"]
verbs: ["get"]
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment threadmanifest/generate.go
A MethodNotAllowed no longer aborts. The authorization review added in
the previous commit ran for every refusal isPatchNotAllowed accepts, so a
405 against credentials that may patch was read as "not a permission
problem" and surfaced. A 405 says the API server takes no patch for the
resource whoever is asking, so being entitled to send one changes
nothing; only a 403 is ambiguous enough to be worth the review.
The least-privilege role could not read the release. Without an explicit
--revision helm lists the storage backend to find the newest one, so
`get` on Secrets and ConfigMaps is not enough - the release storage needs
`list` as well, while the kinds a chart renders still only need `get`.
The HELM_DIFF_THREE_WAY_MERGE_MODE examples did not enable the three-way
merge, so they demonstrated an ordinary two-way diff. Now that the
variable is deliberately ignored unless the merge is on, they have to
pass --three-way-merge to show anything. The README copy of the example
block is maintained by hand rather than by scripts/gen-readme.sh, which
only regenerates the flag tables, so it had drifted from the binary.
The README also documents that client mode can be quieter than the truth
in one place: the restoration copies back every live-only field a
wholesale replacement removed, and nothing local can tell a value the API
server defaulted from one somebody set by hand. A NetworkPolicy port with
a hand-added endPort keeps it, where the upgrade would drop it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@oreonl

Copy link
Copy Markdown
Author

All five points from this review are addressed in d8d73ff. The four suppressed ones have no threads, so answering them here.

MethodNotAllowed aborting instead of falling back (manifest/generate.go:233) — correct, and a regression from the SelfSubjectAccessReview I added in ca059ad: the review ran for every refusal isPatchNotAllowed accepts, so a 405 against credentials that may patch was surfaced rather than falling back. Reproduced (patchDenied(405, canPatch=true) returned false) and fixed by restricting the disambiguation to Forbidden — a 405 says the API server takes no patch for that resource whoever is asking, so entitlement changes nothing. Covered by TestClientSideFallbackAlwaysFallsBackOnMethodNotAllowed.

Release storage needs list (README.md:276, :297, :299) — confirmed in the code: helmGetArgs omits --revision when the revision is 0 (cmd/helm.go:141-145), and helm's Secrets driver Query does a List to find the newest revision, so get alone fails before any resource is read. The least-privilege example now grants get, list on configmaps/secrets for the release storage and keeps the rendered kinds at get, with the reason spelled out in the surrounding prose.

The env var examples did not enable the three-way merge (README.md:404, cmd/upgrade.go:170) — right, and self-inflicted: making HELM_DIFF_THREE_WAY_MERGE_MODE deliberately ignored unless the merge is on invalidated my own examples. Both now pass --three-way-merge. Worth flagging for future edits that scripts/gen-readme.sh regenerates only the flag tables, so the README's copy of the example block is hand-maintained and had drifted from the binary; I diffed the two to confirm they match again.

The remaining inline comment, about live-only drift inside a replaced atomic list, is answered on its own thread. I have left that one open, since it is a design trade-off rather than something I have fixed.

For what it is worth while the checks are still pending: golangci-lint v2.12.2 against Go 1.26.4 — the pairing lint.yaml pins via go-version-file — reports 0 issues, and go test ./... passes.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Local normalization can discard newer Kubernetes fields, while explicit-null restoration can hide real drift corrections.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

manifest/generate.go:365

  • Decoding into the compiled Go type silently discards fields unknown to this client-go version. If helm-diff runs against a newer cluster and a chart adds a newly supported field to an otherwise known built-in kind, the local merge contains the field here, this round-trip removes it, and the restoration pass cannot recover it because it is absent from the live object. Client mode then omits a real upgrade change; preserve unknown fields from the merged object during normalization or avoid typed normalization when the typed schema does not cover the object.

manifest/generate.go:474

  • This treats an explicit null exactly like a missing key, but null can be the chart-owned desired value for fields that are not defaulted. For example, if both manifests render runtimeClassName: null while the live Pod template drifted to runtimeClassName: gvisor, the three-way patch clears the field and normalize omits it; this condition then restores gvisor, hiding a correction that the real upgrade makes. Use presence-aware lookups and restrict null restoration to fields known to be server-defaulted rather than restoring every explicit-null field.
 if originalMap[key] == nil && modifiedMap[key] == nil {
mergedMap[key] = liveValue
  • Files reviewed: 7/7 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@oreonl@yxxhero