fix(scripts): stop early-exit pipe consumers from aborting or misbranching

[LukasWodka]

The remaining client sites from the fleet-wide audit of this hazard class (tracebloc/backend#1778). Follows #678, which fixed the one live instance in docker/k3s-cuda/build.sh; these are the latent ones in the same repo. All mechanical, all the same transform: capture the producer, then slice the captured value with a here-string, so the producer always runs to completion.

The shape

producer | head -N / | grep -q / | awk '…exit' under set -euo pipefail. The consumer closes the pipe early, the producer dies of SIGPIPE, pipefail makes the pipeline status 141, and errexit either aborts the script before its own diagnostics run, or — worse, inside an if — reads 141 as "no match" and silently takes the wrong branch.

Sites

file	consumer	consequence today
scripts/resolve-ingestor-digest.sh:152	awk '/^Digest:/ … exit}'	worst shape in the repo — awk exits at ~line 3, so essentially the whole producer lands after the close. Aborts before the "could not resolve a digest" diagnostic and the multi-arch sanity check
scripts/lib/provision.sh:101	head -4	aborts _report_create_failure before warn/hint print why provisioning failed — the function's entire purpose
scripts/lib/preflight.sh:107	awk … exit	mount on a container-heavy host genuinely exceeds 64 KB → 141 aborts the installer inside preflight with no message
scripts/lib/summary.sh:69,73,84,89	4 × grep -q	inside if → misclassified CLIENT_STATE: the user gets "starting" instead of the bad-creds / image-pull-CA / crash remedy
docs/migration-tools/migrate-tenant.sh:242	head -25	migration dies mid-verification, skipping the mysql table dump and the remaining checks
scripts/lib/setup-linux.sh:93-94	2 × grep -q .	inside if → drops the KMODS_REBOOT_REQUIRED hint
.github/workflows/release-helm-chart.yaml:356,358	grep -q, head -25	only the diagnostic dump is lost (the step fails either way)

A correction to the audit's severity model

Writing the regression tests turned up something that changes how these should be read, and I got it wrong on the first attempt.

For grep -q sites the hazard depends on where the match is, not just on producer size. If the matching line is near the end, grep must read the whole stream to find it, never closes early, and no SIGPIPE occurs. Only a match before the producer finishes triggers it.

My first two regression tests appended the matching line after 64 KB of noise — and passed against the unfixed code. They were vacuous. Moving the match to the front made them fail against the old summary.sh and pass against the new one. Worth stating plainly because it means "producer > 64 KB" alone is not the trigger for the grep -q sites; head -N and awk … exit sites close early unconditionally and are the strictly worse shape.

Behaviour-preserving details worth checking in review

• provision.sh: head -4 <<<"$errmatches" on an empty capture yields an empty substitution (trailing newlines stripped), so [[ -n "$errline" ]] still takes the else-branch exactly as before.
• setup-linux.sh: the short-circuit is preserved — neither find runs unless $missing is non-empty — and the nested if keeps the block's exit status 0 when the condition is false, so it can't trip errexit as a trailing [[ … ]] && … would.
• migrate-tenant.sh: guarded so nothing is printed when no PVC block matches, rather than the blank line a bare here-string would emit.
• summary.sh: one shared comment above _diagnose_not_ready instead of four copies of the same note.
• resolve-ingestor-digest.sh:156 (docker manifest inspect --verbose | grep -m1 | sed) is left as-is: grep -m1 closes early on the fleet's largest producer, but || true outside the substitution already swallows the status and the value is captured correctly. sed drains, so there's no second early-exit.

Test plan

Two new regression tests in summary.bats, following the house pattern already established by cluster.bats:282, check-facts.bats:159-183 and gpu-amd.bats:71-93 for this exact bug class.

summary 28 tests, 0 failed
preflight 124 tests, 0 failed
setup-linux 133 tests, 0 failed
install-bootstrap 23 tests, 0 failed
check-drift 22 tests, 0 failed
check-facts 14 tests, 0 failed
cluster 85 tests, 0 failed
diagnose 10 tests, 0 failed

Plus the full scripts/tests/*.bats suite (result appended in a comment below).

Mutation-checked: against the pre-change summary.sh, both new tests fail (not ok 4, not ok 5); with the fix, 28/28 pass.

Also verified: bash -n on all six changed shell files, yaml.safe_load on the workflow, and shellcheck --severity=warning — the only findings are 3 + 2 pre-existing SC2034s on untouched lines (identical counts on develop; CI gates libs at --severity=error for exactly that reason).

---

Note

Medium Risk
Touches installer, provisioning, preflight, and migration shell paths that run under set -euo pipefail; behavior is intended to be preserved, but incorrect capture/slicing could still hide diagnostics or misclassify client readiness.

Overview
Closes the remaining backend#1778 sites where an early-closing pipe consumer (head, grep -q, awk … exit) made producers die with SIGPIPE under pipefail, either aborting the script or silently taking the wrong if branch.

Applies the same mechanical transform everywhere: capture the producer, then slice via a here-string so the producer always finishes. Highest-impact sites are resolve-ingestor-digest.sh (digest resolve aborted before its error path), _report_create_failure in provision.sh (failed before printing why create failed), and _diagnose_not_ready in summary.sh (could downgrade crash/bad-creds/image-pull to “starting”).

Also hardens preflight.sh (mount fstype probe), setup-linux.sh (kernel-module reboot hint), and migrate-tenant.sh (manifest keep-annotation check). Adds two summary.bats regression tests that put the match before >64KB of noise so the SIGPIPE case is actually exercised.

^{Reviewed by Cursor Bugbot for commit 3dd60d0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

Full scripts/tests/*.bats suite on this head: 923 tests, 0 failed.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 36a8df5. Configure here.}

[LukasWodka]

Scope correction — dropped the release-helm-chart.yaml change (42a2351)

I removed one of the seven sites from this PR after verifying it more carefully. The table in the description listed .github/workflows/release-helm-chart.yaml:356,358. That step is not vulnerable:

• no shell: key on the step
• no defaults.run.shell at workflow or job level
• no set -o pipefail in the body

So Actions runs it as bash -e {0} — errexit on, pipefail off. Without pipefail the pipeline's status is the consumer's, so a SIGPIPE'd producer can't change the outcome. There was no bug, and churn on a release workflow isn't free.

The six shell-script sites stand, and I checked each one rather than assuming:

file	pipefail comes from
scripts/resolve-ingestor-digest.sh	own set -euo pipefail
docs/migration-tools/migrate-tenant.sh	own set -euo pipefail
scripts/lib/preflight.sh	sourced by install-k8s.sh (L61) and install.sh — both set -euo pipefail
scripts/lib/provision.sh	sourced by install-k8s.sh (L82) and install.sh
scripts/lib/setup-linux.sh	sourced by install-k8s.sh (L66) and install.sh
scripts/lib/summary.sh	sourced by install.sh and check-facts.sh

Also in this push: scripts/manifest.sha256 regenerated (1d5b131)

The Static-analysis failure was mine — the bootstrap verifies every sub-script it fetches against that manifest before the privileged steps, so editing four libs without regenerating it makes the installer refuse its own scripts. Regenerated with scripts/gen-manifest.sh; the four hashes match exactly what CI computed.

On the PATH persist — alpine:3 failure — not from this PR

✖ zsh install.sh (SHELL=zsh) exited 1, caused by curl failing TLS verification while fetching SHA256SUMS ("failed to download SHA256SUMS — release may be malformed"). bash and fish both passed in the same job. develop's own Installer-tests run fails the same way (on fedora:latest, ✖ bash), so it is environmental rather than a regression here.

I had first suspected my here-strings broke busybox ash on Alpine — that was wrong: every file changed is #!/usr/bin/env bash, and the job explicitly bootstraps bash (apk add bash; exec bash scripts/tests/path-persist.sh).

[saadqbal]

Clean, careful PR 👍 Same capture-then-here-string transform everywhere, behavior preserved. Verified: the errmatches local-leak was the only real trap and the head commit already scopes it; every new var in a sourced lib (mount_out, mod_this_kernel/any_kernel, errline/errmatches) is local, standalone-script vars are top-level as intended. Reran the mutation check — both new summary.bats tests fail against the old pipe matcher and pass against the fix, so they genuinely exercise the SIGPIPE path. summary 28/0, preflight 124/0, setup-linux 133/0; shellcheck clean apart from the pre-existing SC2034s; manifest hashes match the four changed libs.