feat: make --three-way-merge work without patch permissions - #1063
feat: make --three-way-merge work without patch permissions#1063oreonl wants to merge 7 commits into
Conversation
--three-way-merge computed the merge patch locally but asked the API server to apply it as a dry-run, which needs the `patch` verb on every diffed resource. A read-only account got: cannot patch "x" with kind Deployment: ... is forbidden The patch is now applied locally when the server round-trip is not available, using the same strategic-merge (or JSON merge patch, for custom resources) logic the API server would use. Only `get` is required. A new --three-way-merge-mode flag (env HELM_DIFF_THREE_WAY_MERGE_MODE) selects between: auto server dry-run, falling back to the local merge on Forbidden or MethodNotAllowed. Any other error still aborts, so a genuinely bad patch is not masked. The default. server the previous behaviour; a missing permission is a hard error. client never sends the patch at all. Two post-processing steps keep the local result close to what the API server returns, since client-go does not ship the defaulting functions: * The merged object is round-tripped through its Go type, the way the API server does before it answers. That drops the empty values a manifest spells out but the type omits - `initialDelaySeconds: 0`, `hostNetwork: false`, `sysctls: []` - which would otherwise show up as additions the upgrade does not make. * A field is copied back from the live object when the old and the new release manifest agree about it. The patch replaces `retainKeys` structs and atomic lists as a whole, and a manifest that renders an unset value as an explicit `null` (a bare `replicas:`) reaches the patch as a change rather than a deletion; both wipe values the API server had defaulted in and re-defaults immediately after. Fields the two manifests disagree about are left deleted, because that is a change the chart really makes. manifest.Generate takes variadic options rather than a new parameter, so existing callers keep compiling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A chart that renders an empty collection literally - `rules: []`, the
branch grafana's Role and ClusterRole take when no sidecar is enabled -
disagrees with the cluster about how to write "nothing". Kubernetes
stores objects as protobuf, which cannot tell an empty repeated field
from an absent one, so the API server answers with `rules: null`.
The patch then carries `{"rules":[]}` even though the chart did not
change, and the client-side merge reported:
rules (rbac.authorization.k8s.io/v1/Role/grafana/grafana)
± type change from <nil> to list
server mode shows nothing, because its answer goes back through the same
storage round-trip. The local merge now keeps whichever spelling the
cluster reports whenever the merged and the live value are both empty.
Emptying a collection that actually had entries is still a change and is
still reported.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>There was a problem hiding this comment.
🟡 Changes recommended
auto mode does not currently “fall back per run” (it keeps attempting forbidden PATCHes) and the docs/RBAC wording plus env-var handling need small corrections to match intended behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a new three-way-merge “mode” capability so helm diff upgrade --three-way-merge can work with read-only Kubernetes credentials by falling back (or opting in) to a client-side patch application that approximates the API server’s results.
Changes:
- Introduces
--three-way-merge-mode(auto/server/client) and envHELM_DIFF_THREE_WAY_MERGE_MODE, wiring it throughcmd/upgradeintomanifest.Generate(...)via variadic options. - Implements client-side patch application for three-way-merge, including normalization (round-trip through Go types) and a restoration pass for server-populated/defaulted fields.
- Adds focused unit tests for local merge behavior and CLI flag/env validation; updates README and Copilot instructions.
File summaries
| File | Description |
|---|---|
| README.md | Documents --three-way-merge-mode, behavior tradeoffs, and RBAC guidance. |
| manifest/generate.go | Adds merge-mode options and client-side patch application with post-processing to reduce diff noise. |
| manifest/generate_test.go | New tests covering strategic/merge patch local application and default-restoration behaviors. |
| cmd/upgrade.go | Adds flag/env plumbing and validation; passes merge-mode into manifest generation. |
| cmd/upgrade_test.go | Adds validation coverage for the new flag/env behavior. |
| .github/copilot-instructions.md | Documents the new environment variable. |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
auto mode now falls back for the whole run, as documented. It warned once but kept sending a dry-run PATCH for every remaining resource, so a release of N objects produced N denied requests - N audit log entries and N round-trips - to reach the same conclusion each time. A refused dry-run says something about the credentials rather than about the resource, so the first refusal now moves the rest of the run to the local merge. The decision moved into a clientSideFallback type so it can be tested without a cluster. HELM_DIFF_THREE_WAY_MERGE_MODE is read only when the run actually performs a three-way merge (--three-way-merge, or --take-ownership which implies it). A stale value in the environment could otherwise fail an unrelated `helm diff upgrade` over a mode it was never going to use. The flag stays validated unconditionally: passing it is deliberate, so a typo is worth reporting either way. The README no longer calls a wildcard role "minimal RBAC" - it grants read access to every resource in every API group. It is now described as the blunt option, next to a scoped example and a note that a kind missing from the role fails the diff for that resource. Also fixes the grammar in the two createPatch comments the review flagged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The new validation error messages format the allowed-mode slice with %q, which will render as %!q([]string=...) and should be corrected before merge.
Review details
Suppressed comments (2)
Previously missed (1) — in code that hasn't changed since the last review.
cmd/upgrade.go:199
- The format string uses %q for
manifest.ValidThreeWayMergeModes(a[]string), which will render as%!q([]string=...)and produce an unhelpful validation error. Use%v(or join the slice) when printing the allowed values.
This issue also appears on line 226 of the same file.
cmd/upgrade.go:228
- The env-var validation error formats
manifest.ValidThreeWayMergeModes(a[]string) with%q, which will print as%!q([]string=...). Use%v(or join the slice) so users see the allowed values clearly.
if !slices.Contains(manifest.ValidThreeWayMergeModes, mode) {
return fmt.Errorf("env var %q must be one of %q, but got %q", "HELM_DIFF_THREE_WAY_MERGE_MODE", manifest.ValidThreeWayMergeModes, mode)
}
- Files reviewed: 6/6 changed files
- Comments generated: 0 new
- Review effort level: Lite
oreonl
commented
Sep 3, 2026
Checked this one and I don't think it holds — Actual output from the current branch, for both lines flagged:
So I've left these as they are. |
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core diff computation behavior with non-trivial Kubernetes patch/defaulting semantics, so it warrants final human review despite good test coverage.
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 1
- Review effort level: Lite
Uh oh!
There was an error while loading. Please reload this page.
The marshal of originalObj reported "serializing current configuration", while the live object marshalled a few lines below reported "serializing live configuration". Two different objects, and the first was named after the second, which is misleading when a failure has to be diagnosed. The three messages now say original, target and live. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core three-way-merge behavior with substantial new merge/restoration logic whose correctness and compatibility impact are difficult to fully validate via automated review alone.
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 0 new
- Review effort level: Lite
yxxhero
commented
Sep 3, 2026
@oreonl PTAL |
There was a problem hiding this comment.
🟡 Changes recommended
Local restoration can hide real drift and CRD changes, while automatic fallback can mask admission failures.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 4
- Review effort level: Balanced
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Four defects from review, all confirmed by reproduction before the fix.
restoreMissing copied a live value back whenever the two release
manifests agreed about a key, including when they agreed on an actual
value. normalize drops an explicit `hostNetwork: false` through
omitempty, so a chart pinning false against a cluster that drifted to
true had the drift restored and the correction vanished from the diff -
the opposite of what three-way merge is for. A value is now restored only
where neither manifest carries one, an absent key and a `null` counting
alike, which is all the unset `replicas:` case ever needed.
The restoration pass no longer runs behind a JSON merge patch. That patch
carries only what the chart changed between the two manifests, so it
never prunes a field the cluster populated and there is nothing to put
back. Custom resources are also stored as JSON rather than protobuf,
which keeps `null`, `[]` and `{}` apart, so the empty-value equivalence
was undoing a chart deliberately changing one into another.
auto no longer reads every 403 as a missing patch permission. An
admission webhook or a quota controller can refuse a request the
credentials are entitled to make, and that refusal is exactly what the
upgrade would hit, so falling back locally hid the outcome the diff
exists to predict. A SelfSubjectAccessReview now settles whether patching
is permitted before the refusal is treated as a permission problem; where
the review cannot be made the previous behaviour stands. The fallback is
already once per run, so this costs one extra request.
The README documents a second deviation from server mode: changing an
entry inside an atomic list also drops the server defaults elsewhere in
that entry, because positions in a list the chart rewrote cannot be
matched up safely.
k8s.io/api/authorization/v1 is added to the depguard allowlist, which
golangci-lint requires for the new import.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>There was a problem hiding this comment.
🟡 Changes recommended
Fallback handling and field restoration have correctness gaps, and several usage and RBAC examples are inaccurate.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (5)
Previously missed (3) — in code that hasn't changed since the last review.
README.md:276
- The release-storage requirement is understated here: the default revision is resolved by
helm get manifestwithout--revision, which lists the Helm storage backend to find the newest release. A role with onlygeton Secrets/ConfigMaps therefore fails for the normal invocation; documentlistas required as well.
This issue also appears on line 297 of the same file.
README.md:404
- This generated example does not enable three-way merge. Since the mode environment variable is intentionally ignored unless three-way merge is already active, the command performs a normal two-way diff and does not demonstrate client-side merging.
cmd/upgrade.go:170 - This example does not enable three-way merge. The command only reads this environment variable when
HELM_DIFF_THREE_WAY_MERGE=true(or--three-way-merge/--take-ownership) is already active, so as written it performs the ordinary two-way diff rather than a local merge.
manifest/generate.go:233
MethodNotAllowednever falls back when the authorization review says patching is allowed:isPatchNotAllowedroutes it here, then this check returns false and aborts. That contradicts the newautocontract, which explicitly falls back forMethodNotAllowed. Restrict the authorization disambiguation toForbidden; a 405 indicates the server path is unavailable regardless of RBAC.
if f.canPatch != nil {
if allowed, err := f.canPatch(info); err == nil && allowed {
return false
}
README.md:299
- The least-privilege example cannot read the latest Helm release with the default revision. This code invokes
helm get manifestwithout--revision(cmd/helm.go:139-158), so Helm must list the active Secret/ConfigMap storage backend to find the newest revision; granting onlygetcauses the diff to fail before resource reads. Grantlistto release storage while keeping workload kinds atget.
- apiGroups: [""]
resources: ["configmaps", "secrets", "services", "serviceaccounts"]
verbs: ["get"]
- Files reviewed: 7/7 changed files
- Comments generated: 1
- Review effort level: Balanced
Uh oh!
There was an error while loading. Please reload this page.
A MethodNotAllowed no longer aborts. The authorization review added in the previous commit ran for every refusal isPatchNotAllowed accepts, so a 405 against credentials that may patch was read as "not a permission problem" and surfaced. A 405 says the API server takes no patch for the resource whoever is asking, so being entitled to send one changes nothing; only a 403 is ambiguous enough to be worth the review. The least-privilege role could not read the release. Without an explicit --revision helm lists the storage backend to find the newest one, so `get` on Secrets and ConfigMaps is not enough - the release storage needs `list` as well, while the kinds a chart renders still only need `get`. The HELM_DIFF_THREE_WAY_MERGE_MODE examples did not enable the three-way merge, so they demonstrated an ordinary two-way diff. Now that the variable is deliberately ignored unless the merge is on, they have to pass --three-way-merge to show anything. The README copy of the example block is maintained by hand rather than by scripts/gen-readme.sh, which only regenerates the flag tables, so it had drifted from the binary. The README also documents that client mode can be quieter than the truth in one place: the restoration copies back every live-only field a wholesale replacement removed, and nothing local can tell a value the API server defaulted from one somebody set by hand. A NetworkPolicy port with a hand-added endPort keeps it, where the upgrade would drop it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
oreonl
commented
Sep 4, 2026
All five points from this review are addressed in d8d73ff. The four suppressed ones have no threads, so answering them here.
Release storage needs The env var examples did not enable the three-way merge ( The remaining inline comment, about live-only drift inside a replaced atomic list, is answered on its own thread. I have left that one open, since it is a design trade-off rather than something I have fixed. For what it is worth while the checks are still pending: |
There was a problem hiding this comment.
🔵 Needs a closer look
Local normalization can discard newer Kubernetes fields, while explicit-null restoration can hide real drift corrections.
Review details
Suppressed comments (2)
Previously missed (1) — in code that hasn't changed since the last review.
manifest/generate.go:365
- Decoding into the compiled Go type silently discards fields unknown to this client-go version. If helm-diff runs against a newer cluster and a chart adds a newly supported field to an otherwise known built-in kind, the local merge contains the field here, this round-trip removes it, and the restoration pass cannot recover it because it is absent from the live object. Client mode then omits a real upgrade change; preserve unknown fields from the merged object during normalization or avoid typed normalization when the typed schema does not cover the object.
manifest/generate.go:474
- This treats an explicit
nullexactly like a missing key, butnullcan be the chart-owned desired value for fields that are not defaulted. For example, if both manifests renderruntimeClassName: nullwhile the live Pod template drifted toruntimeClassName: gvisor, the three-way patch clears the field andnormalizeomits it; this condition then restoresgvisor, hiding a correction that the real upgrade makes. Use presence-aware lookups and restrict null restoration to fields known to be server-defaulted rather than restoring every explicit-null field.
if originalMap[key] == nil && modifiedMap[key] == nil {
mergedMap[key] = liveValue
- Files reviewed: 7/7 changed files
- Comments generated: 0 new
- Review effort level: Balanced
What
--three-way-mergecurrently requires thepatchpermission on every diffed resource. It computes the merge patch locally but asks the API server to apply it as a dry-run:A read-only account gets
cannot patch "x" with kind Deployment: ... is forbidden, so the feature is unavailable to exactly the credentials a "show me what this upgrade would do" tool tends to run under.This applies the patch locally when the server round-trip is not available, using the same strategic-merge (or JSON merge patch, for custom resources) logic the API server would use. Only
getis required.Interface
A new
--three-way-merge-modeflag (envHELM_DIFF_THREE_WAY_MERGE_MODE):auto(default)Forbidden/MethodNotAllowed. Any other error still aborts, so a genuinely bad patch is not masked.serverpatchpermission is a hard error.clientautomeans a read-only account works out of the box; it only degrades when the accurate path is unavailable anyway, and says so on stderr.manifest.Generatetakes variadic options rather than a new parameter, so existing callers (helmfile vendors this package) keep compiling.Matching what the server returns
A naive local merge produces a diff full of noise, because the API server does more than apply the patch.
scheme.Scheme.Default()is a no-op in client-go — theSetDefaults_*functions live ink8s.io/kubernetesand are not importable — so the defaults cannot simply be recomputed. Two post-processing steps recover most of the gap:1. Round-trip through the Go type, the way the API server does before it answers. That drops the empty values a manifest spells out but the type omits —
initialDelaySeconds: 0,hostNetwork: false,sysctls: [],supplementalGroups: []— which otherwise appear as additions the upgrade does not make. (Helm'skube.Client.Buildreturns unstructured objects, so these survive from the chart YAML verbatim.)2. Copy a field back from the live object when the old and the new release manifest agree about it. Values disappear from the merged object two ways:
retainKeysstructs (spec.strategy) and atomic lists (ports,volumeClaimTemplates) as a whole, dropping what the server had defaulted into them;null— a barereplicas:, which is what a chart writes for a value it leaves unset — puts the key in both manifests, so it reaches the patch as a change rather than a deletion. That deletes a defaulted value even when the chart did not change at all.The server re-defaults both cases immediately after applying the patch. Locally, a field is restored only when nothing in either manifest asked for it to go. Where the two manifests disagree the deletion is honoured, because that is a change the chart really makes — restoring there would hide real removals (a dropped
nodeSelectorentry looks identical at the JSON level, and the server does not default that back). Values already present in the merged object are never overwritten, so drift between the cluster and the chart is still reported and still corrected.Known deviation from
servermodeWhen the manifests genuinely disagree about a field the API server defaults,
clientmode reports it as removed whereservermode shows it changing to the default — a chart that stops pinningreplicas: 3shows the field going away rather than3→1. The change is reported either way;clientmode just cannot name the value that replaces it. Mutating webhooks and validation are also not applied. All of this is documented in the new README section, along with the minimal RBAC role.Tests
manifest/generate_test.gois new. EightTestLocalMerge_*cases are built from real manifests (unstructured, askube.Client.Buildsupplies them; the live side written as the API server returns it) and cover each symptom above, plus:replicasand hand-edited image in the cluster — still reset to the chart values, i.e. drift detection intact.I checked the tests are not vacuous by reverting each fix independently; each fails without the step it covers.
cmd/upgrade_test.goadds flag/env validation cases.go test ./...,go vetand gofmt pass, and the README flag tables are regenerated withmake readme.Testing done
Built with
make install/helmand run against a real cluster with an account lackingpatch, on a release of ~15 resources (LibreChat and its subcharts). Before the two post-processing steps the client-side output differed from--three-way-merge-mode=serveron defaulted fields; the cases in the tests are taken from that output.🤖 Generated with Claude Code