fix(#554): auto-recover from a pending-* helm wedge in installer and auto-upgrade

[divyasinghds]

Fixes#554.

Problem

A helm process killed mid-operation (Ctrl-C, OOM, host reboot, laptop sleep) leaves the release in a pending-install/pending-upgrade state. The next helm upgrade --install then fails with "another operation is in progress" — exit 1, not 124 — so the existing unwedge hint (guarded by -eq 124) never fires and nothing auto-recovers.

• Site A — installer (scripts/lib/install-client-helm.sh): a prior killed run wedges every subsequent re-run with only a generic "installation failed".
• Site B — auto-upgrade cronjob (client/templates/auto-upgrade-cronjob.yaml): a killed upgrade wedges every hourly tick forever, silently killing security-fix delivery until a human runs helm rollback.

Fix

Installer:

• New _recover_pending_helm_release helper — reads helm status (jq-free), then helm rollback a pending-upgrade/pending-rollback, or helm uninstall a pending-install/uninstalling release, before the helm op.
• Wired into both the normal install and adopt-reconcile paths; fails closed if the wedge can't be cleared rather than marching into it.
• Surfaces the manual unwedge remedy on exit 1 too, not only the 124 timeout; adds --cleanup-on-fail to both upgrades.
• The uninstall branch is data-safe: the chart renders helm.sh/resource-policy: keep on every PVC and helm reads it from the stored manifest, so PVCs survive (per docs/MIGRATIONS.md).

Cronjob:

• Rolls back a pending-upgrade/pending-rollback wedge before retrying.
• Switches --wait → --atomic --cleanup-on-fail so a graceful failure self-reverts (the pre-flight rollback covers the abrupt-SIGKILL case --atomic cannot).

Chart bumped 1.9.28 → 1.9.29 (version + appVersion) so the fix actually publishes; manifest.sha256 regenerated.

Testing

• bash -n + shellcheck clean; embedded cronjob script passes sh -n.
• 113 installer bats tests pass (10 new: helper units, integration wedge-recovery, fail-closed, exit-1 hint).
• 362 helm unittests pass (6 new cronjob-recovery assertions).
• chart-version-guard.sh ✓; gen-manifest.sh --check ✓.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes install/upgrade/reconcile and security-fix auto-upgrade paths with destructive recovery (rollback/uninstall) and Helm flag changes; behavior is heavily tested but mistakes could affect live releases or credentials on adopt paths.

Overview
Fixes #554: a Helm process killed mid-operation leaves releases stuck in pending-* states, so later helm upgrade fails with another operation is in progress (exit 1, not timeout 124) and recovery never ran automatically.

Installer (install-client-helm.sh, assess.sh): Adds _recover_pending_helm_release to read helm status and rollbackpending-upgrade/pending-rollback or uninstallpending-install/uninstalling before install/reconcile; adopt/reconcile uses no-destroy so it won’t uninstall and drop stored credentials. detect_installed_client and reconcile now helm list with --deployed --failed --pending --uninstalling so wedged releases stay visible across Helm 3/4 defaults. The assess gate treats pending wedges as degraded (pending-wedge) instead of fast-path healthy. Upgrades gain --cleanup-on-fail; failure hints mention manual unwedge on exit 1 as well as timeout.

Auto-upgrade cronjob: Pre-upgrade rollback for pending-upgrade/pending-rollback; uninstalling skips the tick cleanly. Replaces --wait with --atomic --cleanup-on-fail.

Chart 1.9.28 → 1.9.29; manifest.sha256 updated; new bats/helm unittest coverage for recovery ordering and fail-closed behavior.

^{Reviewed by Cursor Bugbot for commit b771d56. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7b0b8ca. Configure here.}

[aptracebloc]

@divyasinghds — Approved. Verified against a fresh clone of the head, not just the description.

• R8 manifestshasum -a 256 -c scripts/manifest.sha256 → all 18 files OK (head b771d560); the client/ chart changes correctly sit outside the signed trust root.
• bats:assess.bats 38/38, install-client-helm.bats 119/119 — matches the claimed counts, incl. the SIGPIPE-under-pipefail regression.
• Logic:helm status parsed from a here-string (no early-exit awk pipe → no pipefail wipe), mutating rollback/uninstall deliberately unbounded, _assess_release_pending fails closed (any error → "wedged"), and no-destroy mode refuses the destructive uninstall to protect the stored credential. Sound and internally consistent.
• All 9 Bugbot findings resolved (each with a fix commit + backing test); the last (pipefail status-parse) is on the current head and covered by installer-test 119, though Bugbot hasn't re-run on b771d560.

Note: this supersedes the now-closed #619 — smaller, more focused, adds the assess-gate degrade path + helm-unittest coverage #619 lacked. Code-owner @saadqbal already approved the current head, so it's mergeable.

🤖 Generated with Claude Code