Skip to content

fix(installer): make the absent-key path in _extract_yaml_value reachable - #525

Merged
shujaatTracebloc merged 6 commits into
developfrom
fix/extract-yaml-value-errexit
Aug 5, 2026
Merged

fix(installer): make the absent-key path in _extract_yaml_value reachable#525
shujaatTracebloc merged 6 commits into
developfrom
fix/extract-yaml-value-errexit

Conversation

@LukasWodka

@LukasWodkaLukasWodka commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

_extract_yaml_value (install-client-helm.sh:155) piped into grep. On an absent key grep exits 1; under set -o pipefail that rc propagates out of the pipeline and out of the assignment, so under set -e the function aborts at the assignment — which makes the very next line unreachable in exactly the shape it exists to handle:

line=$(grep -E "^${key}:""$file"2>/dev/null | head -1)# exits 1 when the key is absent
[[ -z"$line" ]] &&return# the "key not found" line — never reached

Latent, not live. All three call sites (lines 209, 651, 652) use the $( ) command-substitution form, which suspends errexit for the function body. But the function's contract is "empty when the key is absent", so a bare call is the natural next refactor — and it would abort the install mid-step.

Fix

|| line="" on the assignment — the house idiom already used at assess.sh:53 and common.sh:308-311. Catching any non-zero also keeps the absent-key path reachable if head -1 ever SIGPIPEs grep (141) — the sibling shape fixed in #522. head -1 kept; this PR is only about the grep-rc=1 abort.

The contract is now written down above the function, so the next reader knows "empty" is the intended answer for both absent-key and unreadable-file, and doesn't have to re-derive it from the callers.

Behaviour change is confined to the absent-key/read-error path. Found-key parsing, quote handling, and _strip_paste_garbage are untouched.

Test plan

New bats case pins the bare-statement call under set -euo pipefail — per #523, asserting the $( ) sites still work proves nothing, since they already did.

Mutation-tested: with || line="" removed, the new case fails (status 1, sentinel never printed); with the fix it passes. So it cannot rot into a no-op.

Verified on bash 3.2.57 / GNU grep:

shapebeforeafter
bare call, absent key, errexit liveexit 1 — aborts, return never runsexit 0, empty, execution continues
v="$(_extract_yaml_value …)", absent keyexit 0, ""unchanged
absent file, bare callexit 1exit 0, empty
found key (clientId: "abc-123")abc-123unchanged

Gates run locally, all green:

  • bats scripts/tests/*.bats670/670 ok, 0 failures
  • shellcheck --severity=warning --shell=bash scripts/lib/install-client-helm.sh → clean, rc=0 (no hits at this severity)
  • bash -n scripts/lib/install-client-helm.sh → parses
  • bash scripts/check-style.shok: style + terminology clean
  • scripts/gen-manifest.sh re-run + scripts/manifest.sha256 committed (R8 "Static analysis" gate). Exactly one hash line changed — install-client-helm.sh, the only edited installer script. Not hand-edited; re-running is idempotent.

Context

Found while fixing the two Bugbot SIGPIPE findings in #522, deliberately left out of that PR as out of scope, and documented in its "Also spotted" section.

Fixes#523

🤖 Generated with Claude Code


Note

Low Risk
Low risk: localized YAML helper hardening in the installer with regression tests; behavior for found keys is unchanged and the fix closes fail-open paths in client detection rather than altering runtime cluster logic.

Overview
_extract_yaml_value in install-client-helm.sh now uses || line="" on the grep assignment (same idiom as assess.sh / common.sh) so a missing YAML key does not abort the installer under set -e and pipefail when the helper is called as a bare statement—not only inside $( ).

The grep | head -1 pipeline is removed: duplicate keys are read with grep alone, then the first line is taken via ${line%%$'\n'*}, avoiding SIGPIPE/pipefail wiping a captured value and weakening detect_installed_client.

The function’s “empty means no value” contract is documented in comments. scripts/manifest.sha256 is updated for the lib change. Bats adds cases for absent-key under set -euo pipefail and duplicate-key first-value behavior.

Reviewed by Cursor Bugbot for commit 25985d8. Bugbot is set up for automated code reviews on this repo. Configure here.

…able
`_extract_yaml_value` piped into `grep`. On an ABSENT key grep exits 1; under
`set -o pipefail` that rc propagates out of the pipeline and out of the
assignment, so under `set -e` the function aborts at the assignment — making
the very next line, `[[ -z "$line" ]] && return`, unreachable in exactly the
shape it exists to handle.
Latent, not live: all three call sites (lines 209, 651, 652) use the `$( )`
command-substitution form, which suspends errexit for the function body. But
the documented contract is "empty when the key is absent", so a bare call is
the natural next refactor — and it would abort the install mid-step.
Fix is the house idiom already used in assess.sh and common.sh
`_chart_version`: `|| line=""` on the assignment. Catching any non-zero also
keeps the path reachable if `head -1` ever SIGPIPEs grep (141), the sibling
shape fixed in #522. Contract written down above the function.
Verified (bash 3.2.57, GNU grep):
- bare call, absent key, errexit live -> before: exit 1 (aborts, `return`
never runs) · after: exit 0, empty output, execution continues
- `v="$(_extract_yaml_value …)"`, absent key -> exit 0, "" (unchanged)
- found-key, quoting, and unreadable-file paths unchanged
Adds a bats case pinning the BARE-statement call under `set -euo pipefail`.
Mutation-tested: it fails against the unfixed function, so it cannot rot into
a no-op. Regenerated scripts/manifest.sha256 (R8 gate).
Fixes#523
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LukasWodkaLukasWodka self-assigned this Jul 31, 2026
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

bugbot run

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit ce96a21. Configure here.

LukasWodkaand others added 2 commits July 31, 2026 16:41
Actions dispatched nothing for this PR: 0 runs on the branch 10 minutes after
open, while a sibling PR opened 3 minutes later got all 7. Not a paths/types
filter (standard-checks + chart-version-guard have no paths filter and also
did not fire), not a draft, not an incident (status green), and PR head ==
remote head == local head. GitHub-side miss on the open event; `synchronize`
re-dispatches all six gating workflows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…alue-errexit
# Conflicts:
#	scripts/manifest.sha256
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

bugbot run

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit b5b77e3. Configure here.

@LukasWodka
LukasWodka requested review from shujaatTracebloc and removed request for saadqbalAugust 2, 2026 15:13
aptracebloc
aptracebloc previously approved these changes Aug 4, 2026

@aptraceblocaptracebloc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LukasWodkaApproved. Clean minimal fix for the pipefail/set -e abort class. I ran it and confirmed all four quadrants:

  • bare call + fixed → reaches the not-found path ✓
  • bare call + reverted || line=""aborts before the empty-check (the bug) ✓
  • $()-wrapped + reverted → does NOT abort (rc 0, empty value) — so your "latent" framing is empirically correct ✓
  • Verified all three call sites (lines 222, 664, 665) are $()-wrapped, so no current caller hits it — this is defensive hardening + making the documented not-found contract hold for any caller, not a live-abort fix.

Other checks: manifest hash matches (R8 intact), parses, and the single || line="" correctly covers both the grep-exit-1 (absent key) and the head SIGPIPE-141 (#522 sibling) shapes.

The standout is the test: it exercises the bare call under set -euo pipefail (the only shape that exhibits the bug) and explicitly notes that asserting the $()-wrapped call would prove nothing — which my own replication confirms. Reaching the sentinel is the proof. Uses the house || var="" idiom (same as assess.sh / _chart_version), manifest kept in lockstep.

No blocking issues — latent today, so no urgency, but correct and worth landing to close the recurring footgun before a future bare caller trips it. Touches your code-owned install-client-helm.sh + manifest.sha256 (R8), so Asad's code-owner review is the backstop; this is the second set of eyes.

🤖 Generated with Claude Code

…alue-errexit
# Conflicts:
#	scripts/manifest.sha256
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

Resolved the develop conflict: the only collision was scripts/manifest.sha256 (#591's install-k8s.ps1 hash landed on develop after this branch's line for install-client-helm.sh). Resolved by regenerating with scripts/gen-manifest.sh from the merged tree — every line re-verified against the actual file hashes. install-client-helm.bats passes (80/80, 0 failures).

Reviewed the substance while here: the || line="" guard is the right shape (house idiom from _chart_version), and the new bare-call test under set -euo pipefail pins the previously-unreachable path — and its assertions are already || return 1-hardened, so it also passes #527's incoming bats-hygiene gate as-is.

Note: the conflict-resolution push dismissed the earlier approval (stale-review dismissal) — needs a re-approve.

bugbot run

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9d0a40f. Configure here.

Comment threadscripts/lib/install-client-helm.sh Outdated
…ep its value (Bugbot, #525)
The first fix (`grep | head -1 || line=""`) traded one failure for another:
on a DUPLICATE key, head exits after the first line and SIGPIPEs grep (141);
under pipefail the `|| line=""` fallback then wiped the successfully captured
value, so detect_installed_client could miss a clientId and fail open toward
overwrite. Capture every match and take the first line in the shell
(`${line%%$'\n'*}`) — no downstream consumer, so grep's rc is 1 exactly when
there is no match, which is the one case the fallback exists for. Regression
test pins the duplicate-key bare-call shape; manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

Fixed the Bugbot finding: dropped head from the pipeline entirely — grep captures every match and the shell takes the first line (${line%%$'\n'*}), so a duplicate key can no longer SIGPIPE grep and have || line="" wipe a successfully captured value. grep's rc is now 1 exactly when there's no match — the one case the fallback exists for. Regression test pins the duplicate-key bare-call shape under set -euo pipefail; manifest regenerated; 81/81 bats green.

bugbot run

@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

Merged develop (now containing #592). Zero conflicts this time — #592 and this PR change different manifest lines, and every merged manifest line re-verified against the actual file hashes. 81/81 bats green.

bugbot run

@shujaatTraceblocshujaatTracebloc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Correct fix: line=$(grep …) || line="" handles the errexit abort on an absent key (#523), and dropping | head -1 for line="${line%%$'\n'*}" fixes the SIGPIPE-wipes-value bug on a duplicate key (fail-open toward overwrite). Tests exercise the real bare-call set -euo pipefail shape and enforce with || return 1. CI green, Bugbot clean, 0 unresolved. LGTM.

@shujaatTracebloc
shujaatTracebloc merged commit 423c1bd into developAug 5, 2026
93 of 96 checks passed
@shujaatTracebloc
shujaatTracebloc deleted the fix/extract-yaml-value-errexit branch August 5, 2026 06:43
shujaatTracebloc pushed a commit that referenced this pull request Aug 5, 2026
…eqs hang (#593)
* fix(ci): bound the two unbounded network waits behind the ubuntu Prereqs hang
Three times on 2026-08-04 (#525, #592) the "Prereqs — ubuntu:*" matrix jobs
died at the 20-minute job timeout with nothing in the log but "Installing
Docker…", and once more failed in 20 seconds with a registry-1.docker.io
timeout (exit 125). Two unbounded waits, one per layer:
- Workflow: `docker run` pulls the distro image implicitly with no timeout,
so Hub connectivity trouble either failed fast (exit 125) or stalled the
whole job. Both container-matrix jobs (distro-prereqs, path-persist) now
pre-pull with three bounded attempts (timeout 300 + backoff) and an honest
"runner-to-registry connectivity, not this PR" error.
- setup-linux.sh: the get.docker.com convenience script's internal
apt/download.docker.com fetches carry no timeout, so a stalled connection
hung silently behind the spinner. The run is now bounded at 10 minutes
(healthy installs take 1-3) and fails with a clear stalled-download message
telling the operator to re-run; the fetch of the script itself already had
retry + curl_secure timeouts. Same shape as the existing dpkg-lock and
kubectl-fetch bounds.
New bats test pins the timeout bound on the get.docker.com branch (hardened
with || return 1 for the incoming #527 hygiene gate). Manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(ci): distinguish a stall from a real failure; keep the pull budget small (Bugbot ×2)
- setup-linux.sh: `if ! spin_cmd …; then error "stalled 10 minutes"` fired on
ANY failure, mislabelling a fast real apt/script error as a stall — and it
bypassed the existing spin_cmd_bounded helper, which returns 124 only on the
deadline and tails the log on every failure. Switched to it: rc 124 gets the
stalled-download message, any other rc gets an honest install-failed message
pointing at the log tail. Harness gains a default spin_cmd_bounded mock; the
bats test now pins the helper + its 600s bound.
- installer-tests.yaml: three timeout-300 attempts + backoff could eat ~16 of
the job's 20 minutes, so a late-succeeding pull just moved the death from
the pull to the install. Bounds resized (3 × timeout 90, 10/20s backoff,
~5.5 min worst case) so the job keeps most of its budget; a healthy pull
takes seconds.
Manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(installer): prepare-host gets prepare-host re-run advice (Bugbot)
The new get.docker.com stall/failure errors always said "re-run the
installer" — but with TB_PREPARE_HOST_MODE set that points an admin at a
full provision as themselves, the exact outcome prepare-host exists to
prevent. Pick the re-run verb by mode, matching the daemon-check errors
later in the same function. Manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
LukasWodka added a commit that referenced this pull request Aug 5, 2026
…ine resolver (Bugbot)
The auto branch treated a failed ls -A as an empty datadir — on arm64,
--reuse-data commonly leaves a uid-999 mysql dir the host user cannot
list, so the resolver opted the reuse into 8.4 and the format guard then
(correctly) refused the 5.7 datadir: the reuse path never came up. An
unlistable dir now counts as content (mirrors _leftover_data_dirs' fail-
closed stance for the same ownership case), with a chmod-000 regression
test. Rebased over #593/#527/#525 (manifest regenerated; my bats
negations now carry the #527 '|| return 1' enforcement idiom).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LukasWodka added a commit that referenced this pull request Aug 5, 2026
…ine resolver (Bugbot)
The auto branch treated a failed ls -A as an empty datadir — on arm64,
--reuse-data commonly leaves a uid-999 mysql dir the host user cannot
list, so the resolver opted the reuse into 8.4 and the format guard then
(correctly) refused the 5.7 datadir: the reuse path never came up. An
unlistable dir now counts as content (mirrors _leftover_data_dirs' fail-
closed stance for the same ownership case), with a chmod-000 regression
test. Rebased over #593/#527/#525 (manifest regenerated; my bats
negations now carry the #527 '|| return 1' enforcement idiom).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LukasWodka added a commit that referenced this pull request Aug 6, 2026
…r format guard (backend#723 PR-2) (#597)
* feat(mysql): A2 engine split — 8.4 opt-in for fresh installs + datadir format guard (backend#723 PR-2)
Chart: mysql-format-guard init container fails fast (with an actionable
message) when the engine major and datadir format disagree — 8.4 over a
5.7-format datadir and 5.7 over an 8.x one are both refused before mysqld
CrashLoops; the 8.0 transit hop and custom digest pins stand down.
tracebloc.mysqlEngineMajor derives the expected engine (digest-wins,
mirroring tracebloc.image); the 5.7 digest literal is CI-pinned to the
values default. Default render changes by exactly the guard.
Installer (A2, decision 2026-08-05): _resolve_mysql_engine picks the engine
for the generated values — explicit TB_MYSQL_ENGINE wins; a previous 8.4
opt-in is sticky; any existing release or real datadir content pins 5.7;
only a fresh arm64 install auto-selects 8.4 (native multi-arch instead of
amd64 emulation). amd64 fresh installs stay 5.7 for now (soak first).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(installer): fail CLOSED on an unlistable mysql datadir in the engine resolver (Bugbot)
The auto branch treated a failed ls -A as an empty datadir — on arm64,
--reuse-data commonly leaves a uid-999 mysql dir the host user cannot
list, so the resolver opted the reuse into 8.4 and the format guard then
(correctly) refused the 5.7 datadir: the reuse path never came up. An
unlistable dir now counts as content (mirrors _leftover_data_dirs' fail-
closed stance for the same ownership case), with a chmod-000 regression
test. Rebased over #593/#527/#525 (manifest regenerated; my bats
negations now carry the #527 '|| return 1' enforcement idiom).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(bats): harden the 11 new engine-resolver assertions per the #527 hygiene guard
bats-hygiene's scanner requires every standalone bracket assertion in an
@test body to end in '|| return 1'; the resolver tests added on this
branch predated rebasing onto that guard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(tests): restore the brace the #604 merge seam ate (last mirror test)
git hoisted the shared closing brace out of the conflict region during the
rebase onto #604; the file then died at parse (1 of 88 tests ran).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(chart): the format guard honors global.imageRegistry (Bugbot)
Semantic rebase conflict with #604: every other image include gained the
mirror dig while the guard (written pre-#604, merged clean textually)
kept a hardcoded docker.io — on mirrored/air-gapped edges the always-on
guard alone would ImagePullBackOff and block mysql on exactly the fleets
#604 serves. Same dig expression now + a mirror re-home pin test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(chart): bump 1.9.20 -> 1.9.21, as this PR changes chart content
My earlier merge resolution took develop's 1.9.20 verbatim, reasoning from the
release train's rule: v1.9.20 is untagged, and the train's version_preflight
only refuses when the version is ALREADY released, so one bump covers a whole
release cycle. That reasoning is correct for the train and wrong for this repo.
client/scripts/chart-version-guard.sh enforces a stricter rule for a
repo-specific reason: chart content reaches installs only via a NEW chart
version, because a Helm repo publishes on version change. An unbumped
template/values edit therefore either reaches nobody or overwrites an
already-published version. Both have happened here - the perIngestionTables
block shipped dark in PR #472, and ingestor-0.2.0.tgz was overwritten 5x
between 2026-05-20 and 2026-07-29.
This PR changes client/templates/** and client/values.yaml, so it needs its
own version rather than riding develop's. v1.9.21 is untagged.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
divyasinghds added a commit that referenced this pull request Aug 11, 2026
…pe (Bugbot #649)
The prior `|| _status=""` guard fixed the set -e abort but RE-INTRODUCED the
SIGPIPE-under-pipefail wipe the codebase already learned about (_extract_yaml_value,
#525): on a WEDGED release awk's `exit` SIGPIPEs helm (141) as it writes the rest
of the status body, so under `set -o pipefail` the pipe is non-zero and the guard
wiped the correctly-parsed "pending-upgrade" — recovery then silently no-op'd and
re-runs still hit "another operation is in progress". Defeats the whole fix.
Capture the full `helm status` first (guarded), then parse it from a here-string
(no pipe, nothing can signal) — the same idiom _extract_yaml_value uses. Added a
regression test: a wedged status with a >64KB body under set -o pipefail must
still trigger rollback (fails on the old code, passes now). 119 installer + 38
assess bats pass; manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
divyasinghds added a commit that referenced this pull request Aug 11, 2026
…auto-upgrade (#649)
* fix(#554): auto-recover from a pending-* helm wedge in installer and auto-upgrade
A helm process killed mid-operation (Ctrl-C, OOM, host reboot, laptop
sleep) leaves the release in a pending-install/pending-upgrade state.
The next `helm upgrade --install` then fails with "another operation is
in progress" — exit 1, not 124 — so the existing unwedge hint (guarded
by `-eq 124`) never fires and nothing auto-recovers. In the auto-upgrade
cronjob this silently kills security-fix delivery: every hourly tick
fails forever until a human runs `helm rollback`.
Installer (scripts/lib/install-client-helm.sh):
- add _recover_pending_helm_release: read `helm status` (jq-free) and
`helm rollback` a pending-upgrade/pending-rollback, or `helm uninstall`
a pending-install/uninstalling release, before the helm op runs.
- wire it into both the normal install and the adopt-reconcile paths;
fail closed if the wedge can't be cleared rather than marching into it.
- surface the manual unwedge remedy on exit 1 too, not only the 124
timeout; add --cleanup-on-fail to both upgrades.
- the uninstall branch is data-safe: the chart renders
helm.sh/resource-policy: keep on every PVC and helm reads it from the
stored manifest, so PVCs survive (docs/MIGRATIONS.md).
Auto-upgrade cronjob (client/templates/auto-upgrade-cronjob.yaml):
- roll back a pending-upgrade/pending-rollback wedge before retrying.
- switch --wait to --atomic --cleanup-on-fail so a graceful failure
self-reverts (the pre-flight rollback covers the abrupt-kill case
--atomic cannot).
Tests: 10 new installer bats cases (helper units, integration
wedge-recovery, fail-closed, exit-1 hint) and 6 new cronjob unittest
assertions. Chart bumped 1.9.28 -> 1.9.29 so the fix publishes;
manifest.sha256 regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(#554): make wedge recovery visible to the guards and bound its read (Bugbot #649)
Bugbot review of the pending-* wedge recovery found three issues:
- Adopt recovery skipped pending releases (High): _reconcile_adopted_client
discovered the release with `helm list -A`, which on Helm 3 shows only
deployed releases unless states are named. A release wedged in pending-*
was invisible, so adopt fell through to a password prompt it can't satisfy.
- One-client guard blind to a wedge (High): detect_installed_client enumerated
with the same bare `helm list -A`, so a foreign client wedged in pending-*
was unseen — a re-run with a different clientId could overwrite it once
recovery cleared the wedge (the wedge itself used to block that overwrite).
- Recovery read unbounded (Medium): _recover_pending_helm_release ran an
unbounded `helm status`, so a wedged/unreachable API could hang a headless
run before the bounded upgrade.
Fixes:
- both enumeration sites now pass `--deployed --pending --failed`, so a wedged
or failed client stays visible to the adopt discovery and the one-client guard.
- bound the `helm status` READ with _bounded (the installer's timeout(1) probe
wrapper); it gates the rest, so a timeout yields empty status -> no-op -> the
bounded upgrade surfaces the API failure. The mutating rollback/uninstall are
deliberately NOT wrapped in a kill-based bound (SIGKILLing them midway would
recreate the wedge); uninstall stays bounded by helm's own --wait --timeout.
Tests: +1 guard regression (a DIFFERENT client wedged in pending-* is still
seen and blocked, asserting --pending is enumerated); the recovery unit +
integration tests override _bounded to reach the helm mock. manifest.sha256
regenerated. 114 installer bats pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(#554): don't uninstall (dropping the adopted credential) on the reconcile path (Bugbot #649)
Bugbot: making pending-* releases visible to the adopt discovery (the previous
commit) newly exposed a credential-loss path. On a re-run after a killed FIRST
install, _reconcile_adopted_client now finds the pending-install release,
_recover_pending_helm_release uninstalls it (a never-deployed rev can't be
rolled back), then the reconcile runs `helm upgrade --reuse-values` (no
--install) -> release-not-found -> error. The adopted credential lived only in
that release (the account password is write-only on the backend), so it's gone.
Fix: add a no-destroy mode to _recover_pending_helm_release. The reconcile/adopt
path passes it: rollback (non-destructive, keeps --reuse-values valid) is still
performed, but the destructive uninstall branch is REFUSED and returns non-zero,
so the caller fails closed with a manual remedy instead of silently destroying
the sole copy of the credential. The normal install path keeps full mode — it
writes a fresh values file with the just-verified credential, so uninstalling a
pending-install there loses nothing.
Tests: +3 no-destroy unit cases (pending-install/uninstalling refused without an
uninstall; pending-upgrade still rolls back). manifest regenerated. 117 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(#554): degrade a pending-* wedge in assess so recovery runs (Bugbot #649)
detect_installed_client now enumerates pending releases, so assess sees a
pending-upgrade release whose prior revision is still Ready and would classify
the machine healthy — handing off before install_client_helm runs recovery, so
the wedge stays dead. Add _assess_release_pending (bounded, jq-free) and degrade
to the normal flow on a wedge. +2 assess tests; existing classify tests mock the
new probe. manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(#554): enumerate uninstalling releases too, version-independently (Bugbot #649)
The installer pins Helm v4.2.3, whose `helm list` default lists ALL statuses;
Helm 3 defaults to deployed-only. Naming --deployed --failed --pending made the
listing version-independent but dropped `uninstalling` — a state
_recover_pending_helm_release already handles — so on Helm 4 a foreign client
stuck mid-uninstall became invisible to the one-client guard, re-opening the
overwrite fail-open. Add --uninstalling to both enumeration sites and to assess's
wedge probe, so the full deployed/failed/pending-*/uninstalling set is named
everywhere regardless of the helm default. manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(#554): comprehensive audit — fail-closed assess probe, errexit guard, real adopt remedy, cronjob uninstalling (Bugbot #649)
One pass instead of round-by-round. Fixes the two open Bugbot findings plus two
issues an adversarial self-audit surfaced first:
- assess `_assess_release_pending` failed OPEN: a helm error/timeout read as
"no wedge" and let the machine fast-path to healthy, skipping recovery. Now
captures the probe's exit code and degrades on any error — matching the
module's "never a false healthy" contract. (Bugbot)
- adopt no-destroy refusal printed only `helm rollback`, which cannot clear
pending-install/uninstalling and never said how to keep the write-only
credential. The helper now prints the credential-safe path (get values ->
uninstall -> re-run); the caller drops the misleading rollback hint. (Bugbot)
- `_status=$(_bounded helm status | awk)` lacked the house `|| _status=""`
errexit guard: safe today only because both callers use `if !` (which masks
set -e), a latent trap on any future bare call. Guarded. (audit)
- cronjob ignored `uninstalling`, so a killed `helm uninstall` failed every tick
with an opaque "another operation is in progress". Now skips cleanly with an
actionable log line. (audit)
Tests: +1 assess (probe fails closed on helm error), +1 installer (bare call
under set -e doesn't abort), no-destroy test asserts the credential-safe remedy,
+1 cronjob unittest (uninstalling handled). 38 assess + 118 installer bats +
cronjob unittest all pass. manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(#554): parse helm status from a here-string, not an early-exit pipe (Bugbot #649)
The prior `|| _status=""` guard fixed the set -e abort but RE-INTRODUCED the
SIGPIPE-under-pipefail wipe the codebase already learned about (_extract_yaml_value,
#525): on a WEDGED release awk's `exit` SIGPIPEs helm (141) as it writes the rest
of the status body, so under `set -o pipefail` the pipe is non-zero and the guard
wiped the correctly-parsed "pending-upgrade" — recovery then silently no-op'd and
re-runs still hit "another operation is in progress". Defeats the whole fix.
Capture the full `helm status` first (guarded), then parse it from a here-string
(no pipe, nothing can signal) — the same idiom _extract_yaml_value uses. Added a
regression test: a wedged status with a >64KB body under set -o pipefail must
still trigger rollback (fails on the old code, passes now). 119 installer + 38
assess bats pass; manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

_extract_yaml_value: absent-key path is unreachable under pipefail + set -e

3 participants

@LukasWodka@shujaatTracebloc@aptracebloc