Skip to content

fix(ci): bound the two unbounded network waits behind the ubuntu Prereqs hang - #593

Merged
shujaatTracebloc merged 3 commits into
developfrom
fix/ubuntu-prereqs-ci-hang
Aug 5, 2026
Merged

fix(ci): bound the two unbounded network waits behind the ubuntu Prereqs hang#593
shujaatTracebloc merged 3 commits into
developfrom
fix/ubuntu-prereqs-ci-hang

Conversation

@LukasWodka

@LukasWodkaLukasWodka commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Three times on 2026-08-04 (#525, #592) the Prereqs — ubuntu:* matrix jobs died at the 20-minute job timeout with nothing in the log but ⠼ Installing Docker… — and a fourth failure took 20 seconds with registry-1.docker.io … Client.Timeout exceeded (exit 125). Every other distro in the matrix passed every time; plain re-runs succeeded. Two unbounded network waits, one per layer:

Changes

Workflow (installer-tests.yaml)docker run pulls the distro image implicitly with no timeout, so Docker Hub connectivity trouble either failed the job fast (exit 125) or stalled it until timeout-minutes killed it. Both container-matrix jobs (distro-prereqs, path-persist) now pre-pull with three bounded attempts (timeout 300 docker pull + backoff) and an honest error: "runner-to-registry connectivity, not this PR. Re-run this job."

setup-linux.sh — the get.docker.com convenience script's internal apt/download.docker.com fetches carry no timeout of their own, so a stalled connection hung silently behind the spinner (the fetch of the script itself already had retry + curl_secure bounds). The run is now bounded at 10 minutes (healthy installs take 1–3) and fails with a clear stalled-download message telling the operator to re-run — the installer resumes safely. Same shape as the existing dpkg-lock (#210) and kubectl-fetch bounds; timeout is coreutils, present on every distro this branch serves.

Test plan

  • New bats test pins the timeout 600 bound on the get.docker.com branch — written || return 1-hardened so it stays green under test(bats): make every assertion enforce — 1240 were advisory #527's incoming bats-hygiene gate.
  • setup-linux.bats 0 failures; actionlint clean; scripts/gen-manifest.sh regenerated (setup-linux.sh is manifested).
  • The Prereqs/PATH persist matrix on this very PR exercises the new pre-pull path live.

Refs the 2026-08-04 incident runs (job ids 92054431018, 92073231942).

🤖 Generated with Claude Code


Note

Low Risk
Changes are timeout/retry wrappers around existing CI and installer paths; no auth, data, or cluster logic changes.

Overview
Addresses CI jobs that sat at Installing Docker… or hit the 20-minute timeout when Docker Hub or get.docker.com stalled—infra failures, not product regressions.

CI (installer-tests.yaml)distro-prereqs and path-persist now pre-pull each matrix image with three bounded attempts (timeout 90 docker pull, backoff) before docker run, so implicit pulls can’t hang the job; failures surface as a clear “runner-to-registry connectivity” message.

Linux installer (setup-linux.sh) — The get.docker.com convenience script run is wrapped in spin_cmd_bounded (600s) instead of unbounded spin_cmd, with distinct errors for timeout (124) vs real install failure and prepare-host-aware re-run guidance.

Tests / supply chain — Bats pins the 600s bound; manifest.sha256 updated for the lib change.

Reviewed by Cursor Bugbot for commit 9b3d81e. Bugbot is set up for automated code reviews on this repo. Configure here.

…eqs hang
Three times on 2026-08-04 (#525, #592) the "Prereqs — ubuntu:*" matrix jobs
died at the 20-minute job timeout with nothing in the log but "Installing
Docker…", and once more failed in 20 seconds with a registry-1.docker.io
timeout (exit 125). Two unbounded waits, one per layer:
- Workflow: `docker run` pulls the distro image implicitly with no timeout,
so Hub connectivity trouble either failed fast (exit 125) or stalled the
whole job. Both container-matrix jobs (distro-prereqs, path-persist) now
pre-pull with three bounded attempts (timeout 300 + backoff) and an honest
"runner-to-registry connectivity, not this PR" error.
- setup-linux.sh: the get.docker.com convenience script's internal
apt/download.docker.com fetches carry no timeout, so a stalled connection
hung silently behind the spinner. The run is now bounded at 10 minutes
(healthy installs take 1-3) and fails with a clear stalled-download message
telling the operator to re-run; the fetch of the script itself already had
retry + curl_secure timeouts. Same shape as the existing dpkg-lock and
kubectl-fetch bounds.
New bats test pins the timeout bound on the get.docker.com branch (hardened
with || return 1 for the incoming #527 hygiene gate). Manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodkaLukasWodka self-assigned this Aug 4, 2026
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

bugbot run

Comment threadscripts/lib/setup-linux.sh
Comment thread.github/workflows/installer-tests.yaml
…t small (Bugbot ×2)
- setup-linux.sh: `if ! spin_cmd …; then error "stalled 10 minutes"` fired on
ANY failure, mislabelling a fast real apt/script error as a stall — and it
bypassed the existing spin_cmd_bounded helper, which returns 124 only on the
deadline and tails the log on every failure. Switched to it: rc 124 gets the
stalled-download message, any other rc gets an honest install-failed message
pointing at the log tail. Harness gains a default spin_cmd_bounded mock; the
bats test now pins the helper + its 600s bound.
- installer-tests.yaml: three timeout-300 attempts + backoff could eat ~16 of
the job's 20 minutes, so a late-succeeding pull just moved the death from
the pull to the install. Bounds resized (3 × timeout 90, 10/20s backoff,
~5.5 min worst case) so the job keeps most of its budget; a healthy pull
takes seconds.
Manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

Both findings fixed:

  1. Stall error masked real failures — switched to the existing spin_cmd_bounded (which I'd bypassed): rc 124 → the stalled-download message; any other rc → an honest install-failed message pointing at the log tail the helper already prints. Fast real apt/script failures keep their own diagnosis now. Bats test re-pinned on the helper + 600s bound.
  2. Pull retries exhausted the job budget — resized to 3 × timeout 90 with 10/20s backoff (~5.5 min worst case vs ~16), so a late-succeeding pull leaves the 20-minute job most of its budget for the actual install.

bugbot run

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d11047a. Configure here.

Comment threadscripts/lib/setup-linux.sh
The new get.docker.com stall/failure errors always said "re-run the
installer" — but with TB_PREPARE_HOST_MODE set that points an admin at a
full provision as themselves, the exact outcome prepare-host exists to
prevent. Pick the re-run verb by mode, matching the daemon-check errors
later in the same function. Manifest regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

Fixed: the stall/failure errors now pick their re-run advice by mode — prepare-host when TB_PREPARE_HOST_MODE is set, the installer otherwise — matching the daemon-check errors later in the same function.

bugbot run

@shujaatTraceblocshujaatTracebloc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Correct and well-reasoned: bounded+retried docker pull in CI (primes the cache before the implicit docker run pull) and spin_cmd_bounded 600 on the get.docker.com run, with mode-aware error advice (prepare-host vs "the installer") so an admin isn't told to re-run a full self-provision. CI green, Bugbot clean, 0 unresolved. Optional non-blocking follow-up: a test for the rc-124/error-message + prepare-host _rerun branches. LGTM.

@shujaatTracebloc
shujaatTracebloc merged commit 988ef49 into developAug 5, 2026
46 checks passed
@shujaatTracebloc
shujaatTracebloc deleted the fix/ubuntu-prereqs-ci-hang branch August 5, 2026 06:45
LukasWodka added a commit that referenced this pull request Aug 5, 2026
…ine resolver (Bugbot)
The auto branch treated a failed ls -A as an empty datadir — on arm64,
--reuse-data commonly leaves a uid-999 mysql dir the host user cannot
list, so the resolver opted the reuse into 8.4 and the format guard then
(correctly) refused the 5.7 datadir: the reuse path never came up. An
unlistable dir now counts as content (mirrors _leftover_data_dirs' fail-
closed stance for the same ownership case), with a chmod-000 regression
test. Rebased over #593/#527/#525 (manifest regenerated; my bats
negations now carry the #527 '|| return 1' enforcement idiom).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LukasWodka added a commit that referenced this pull request Aug 5, 2026
…ine resolver (Bugbot)
The auto branch treated a failed ls -A as an empty datadir — on arm64,
--reuse-data commonly leaves a uid-999 mysql dir the host user cannot
list, so the resolver opted the reuse into 8.4 and the format guard then
(correctly) refused the 5.7 datadir: the reuse path never came up. An
unlistable dir now counts as content (mirrors _leftover_data_dirs' fail-
closed stance for the same ownership case), with a chmod-000 regression
test. Rebased over #593/#527/#525 (manifest regenerated; my bats
negations now carry the #527 '|| return 1' enforcement idiom).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LukasWodka added a commit that referenced this pull request Aug 6, 2026
…r format guard (backend#723 PR-2) (#597)
* feat(mysql): A2 engine split — 8.4 opt-in for fresh installs + datadir format guard (backend#723 PR-2)
Chart: mysql-format-guard init container fails fast (with an actionable
message) when the engine major and datadir format disagree — 8.4 over a
5.7-format datadir and 5.7 over an 8.x one are both refused before mysqld
CrashLoops; the 8.0 transit hop and custom digest pins stand down.
tracebloc.mysqlEngineMajor derives the expected engine (digest-wins,
mirroring tracebloc.image); the 5.7 digest literal is CI-pinned to the
values default. Default render changes by exactly the guard.
Installer (A2, decision 2026-08-05): _resolve_mysql_engine picks the engine
for the generated values — explicit TB_MYSQL_ENGINE wins; a previous 8.4
opt-in is sticky; any existing release or real datadir content pins 5.7;
only a fresh arm64 install auto-selects 8.4 (native multi-arch instead of
amd64 emulation). amd64 fresh installs stay 5.7 for now (soak first).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(installer): fail CLOSED on an unlistable mysql datadir in the engine resolver (Bugbot)
The auto branch treated a failed ls -A as an empty datadir — on arm64,
--reuse-data commonly leaves a uid-999 mysql dir the host user cannot
list, so the resolver opted the reuse into 8.4 and the format guard then
(correctly) refused the 5.7 datadir: the reuse path never came up. An
unlistable dir now counts as content (mirrors _leftover_data_dirs' fail-
closed stance for the same ownership case), with a chmod-000 regression
test. Rebased over #593/#527/#525 (manifest regenerated; my bats
negations now carry the #527 '|| return 1' enforcement idiom).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(bats): harden the 11 new engine-resolver assertions per the #527 hygiene guard
bats-hygiene's scanner requires every standalone bracket assertion in an
@test body to end in '|| return 1'; the resolver tests added on this
branch predated rebasing onto that guard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(tests): restore the brace the #604 merge seam ate (last mirror test)
git hoisted the shared closing brace out of the conflict region during the
rebase onto #604; the file then died at parse (1 of 88 tests ran).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(chart): the format guard honors global.imageRegistry (Bugbot)
Semantic rebase conflict with #604: every other image include gained the
mirror dig while the guard (written pre-#604, merged clean textually)
kept a hardcoded docker.io — on mirrored/air-gapped edges the always-on
guard alone would ImagePullBackOff and block mysql on exactly the fleets
#604 serves. Same dig expression now + a mirror re-home pin test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(chart): bump 1.9.20 -> 1.9.21, as this PR changes chart content
My earlier merge resolution took develop's 1.9.20 verbatim, reasoning from the
release train's rule: v1.9.20 is untagged, and the train's version_preflight
only refuses when the version is ALREADY released, so one bump covers a whole
release cycle. That reasoning is correct for the train and wrong for this repo.
client/scripts/chart-version-guard.sh enforces a stricter rule for a
repo-specific reason: chart content reaches installs only via a NEW chart
version, because a Helm repo publishes on version change. An unbumped
template/values edit therefore either reaches nobody or overwrites an
already-published version. Both have happened here - the perIngestionTables
block shipped dark in PR #472, and ingestor-0.2.0.tgz was overwritten 5x
between 2026-05-20 and 2026-07-29.
This PR changes client/templates/** and client/values.yaml, so it needs its
own version rather than riding develop's. v1.9.21 is untagged.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@LukasWodka@shujaatTracebloc@divyasinghds