Uh oh!
There was an error while loading. Please reload this page.
fix(ci): bound the two unbounded network waits behind the ubuntu Prereqs hang - #593
Conversation
…eqs hang Three times on 2026-08-04 (#525, #592) the "Prereqs — ubuntu:*" matrix jobs died at the 20-minute job timeout with nothing in the log but "Installing Docker…", and once more failed in 20 seconds with a registry-1.docker.io timeout (exit 125). Two unbounded waits, one per layer: - Workflow: `docker run` pulls the distro image implicitly with no timeout, so Hub connectivity trouble either failed fast (exit 125) or stalled the whole job. Both container-matrix jobs (distro-prereqs, path-persist) now pre-pull with three bounded attempts (timeout 300 + backoff) and an honest "runner-to-registry connectivity, not this PR" error. - setup-linux.sh: the get.docker.com convenience script's internal apt/download.docker.com fetches carry no timeout, so a stalled connection hung silently behind the spinner. The run is now bounded at 10 minutes (healthy installs take 1-3) and fails with a clear stalled-download message telling the operator to re-run; the fetch of the script itself already had retry + curl_secure timeouts. Same shape as the existing dpkg-lock and kubectl-fetch bounds. New bats test pins the timeout bound on the get.docker.com branch (hardened with || return 1 for the incoming #527 hygiene gate). Manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LukasWodka
commented
Aug 4, 2026
bugbot run |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…t small (Bugbot ×2) - setup-linux.sh: `if ! spin_cmd …; then error "stalled 10 minutes"` fired on ANY failure, mislabelling a fast real apt/script error as a stall — and it bypassed the existing spin_cmd_bounded helper, which returns 124 only on the deadline and tails the log on every failure. Switched to it: rc 124 gets the stalled-download message, any other rc gets an honest install-failed message pointing at the log tail. Harness gains a default spin_cmd_bounded mock; the bats test now pins the helper + its 600s bound. - installer-tests.yaml: three timeout-300 attempts + backoff could eat ~16 of the job's 20 minutes, so a late-succeeding pull just moved the death from the pull to the install. Bounds resized (3 × timeout 90, 10/20s backoff, ~5.5 min worst case) so the job keeps most of its budget; a healthy pull takes seconds. Manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LukasWodka
commented
Aug 4, 2026
Both findings fixed:
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d11047a. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
The new get.docker.com stall/failure errors always said "re-run the installer" — but with TB_PREPARE_HOST_MODE set that points an admin at a full provision as themselves, the exact outcome prepare-host exists to prevent. Pick the re-run verb by mode, matching the daemon-check errors later in the same function. Manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LukasWodka
commented
Aug 4, 2026
Fixed: the stall/failure errors now pick their re-run advice by mode — bugbot run |
shujaatTracebloc
left a comment
There was a problem hiding this comment.
Reviewed. Correct and well-reasoned: bounded+retried docker pull in CI (primes the cache before the implicit docker run pull) and spin_cmd_bounded 600 on the get.docker.com run, with mode-aware error advice (prepare-host vs "the installer") so an admin isn't told to re-run a full self-provision. CI green, Bugbot clean, 0 unresolved. Optional non-blocking follow-up: a test for the rc-124/error-message + prepare-host _rerun branches. LGTM.
Uh oh!
There was an error while loading. Please reload this page.
…ine resolver (Bugbot) The auto branch treated a failed ls -A as an empty datadir — on arm64, --reuse-data commonly leaves a uid-999 mysql dir the host user cannot list, so the resolver opted the reuse into 8.4 and the format guard then (correctly) refused the 5.7 datadir: the reuse path never came up. An unlistable dir now counts as content (mirrors _leftover_data_dirs' fail- closed stance for the same ownership case), with a chmod-000 regression test. Rebased over #593/#527/#525 (manifest regenerated; my bats negations now carry the #527 '|| return 1' enforcement idiom). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ine resolver (Bugbot) The auto branch treated a failed ls -A as an empty datadir — on arm64, --reuse-data commonly leaves a uid-999 mysql dir the host user cannot list, so the resolver opted the reuse into 8.4 and the format guard then (correctly) refused the 5.7 datadir: the reuse path never came up. An unlistable dir now counts as content (mirrors _leftover_data_dirs' fail- closed stance for the same ownership case), with a chmod-000 regression test. Rebased over #593/#527/#525 (manifest regenerated; my bats negations now carry the #527 '|| return 1' enforcement idiom). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r format guard (backend#723 PR-2) (#597) * feat(mysql): A2 engine split — 8.4 opt-in for fresh installs + datadir format guard (backend#723 PR-2) Chart: mysql-format-guard init container fails fast (with an actionable message) when the engine major and datadir format disagree — 8.4 over a 5.7-format datadir and 5.7 over an 8.x one are both refused before mysqld CrashLoops; the 8.0 transit hop and custom digest pins stand down. tracebloc.mysqlEngineMajor derives the expected engine (digest-wins, mirroring tracebloc.image); the 5.7 digest literal is CI-pinned to the values default. Default render changes by exactly the guard. Installer (A2, decision 2026-08-05): _resolve_mysql_engine picks the engine for the generated values — explicit TB_MYSQL_ENGINE wins; a previous 8.4 opt-in is sticky; any existing release or real datadir content pins 5.7; only a fresh arm64 install auto-selects 8.4 (native multi-arch instead of amd64 emulation). amd64 fresh installs stay 5.7 for now (soak first). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): fail CLOSED on an unlistable mysql datadir in the engine resolver (Bugbot) The auto branch treated a failed ls -A as an empty datadir — on arm64, --reuse-data commonly leaves a uid-999 mysql dir the host user cannot list, so the resolver opted the reuse into 8.4 and the format guard then (correctly) refused the 5.7 datadir: the reuse path never came up. An unlistable dir now counts as content (mirrors _leftover_data_dirs' fail- closed stance for the same ownership case), with a chmod-000 regression test. Rebased over #593/#527/#525 (manifest regenerated; my bats negations now carry the #527 '|| return 1' enforcement idiom). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(bats): harden the 11 new engine-resolver assertions per the #527 hygiene guard bats-hygiene's scanner requires every standalone bracket assertion in an @test body to end in '|| return 1'; the resolver tests added on this branch predated rebasing onto that guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(tests): restore the brace the #604 merge seam ate (last mirror test) git hoisted the shared closing brace out of the conflict region during the rebase onto #604; the file then died at parse (1 of 88 tests ran). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(chart): the format guard honors global.imageRegistry (Bugbot) Semantic rebase conflict with #604: every other image include gained the mirror dig while the guard (written pre-#604, merged clean textually) kept a hardcoded docker.io — on mirrored/air-gapped edges the always-on guard alone would ImagePullBackOff and block mysql on exactly the fleets #604 serves. Same dig expression now + a mirror re-home pin test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(chart): bump 1.9.20 -> 1.9.21, as this PR changes chart content My earlier merge resolution took develop's 1.9.20 verbatim, reasoning from the release train's rule: v1.9.20 is untagged, and the train's version_preflight only refuses when the version is ALREADY released, so one bump covers a whole release cycle. That reasoning is correct for the train and wrong for this repo. client/scripts/chart-version-guard.sh enforces a stricter rule for a repo-specific reason: chart content reaches installs only via a NEW chart version, because a Helm repo publishes on version change. An unbumped template/values edit therefore either reaches nobody or overwrites an already-published version. Both have happened here - the perIngestionTables block shipped dark in PR #472, and ingestor-0.2.0.tgz was overwritten 5x between 2026-05-20 and 2026-07-29. This PR changes client/templates/** and client/values.yaml, so it needs its own version rather than riding develop's. v1.9.21 is untagged. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Summary
Three times on 2026-08-04 (#525, #592) the
Prereqs — ubuntu:*matrix jobs died at the 20-minute job timeout with nothing in the log but⠼ Installing Docker…— and a fourth failure took 20 seconds withregistry-1.docker.io … Client.Timeout exceeded(exit 125). Every other distro in the matrix passed every time; plain re-runs succeeded. Two unbounded network waits, one per layer:Changes
Workflow (
installer-tests.yaml) —docker runpulls the distro image implicitly with no timeout, so Docker Hub connectivity trouble either failed the job fast (exit 125) or stalled it untiltimeout-minuteskilled it. Both container-matrix jobs (distro-prereqs,path-persist) now pre-pull with three bounded attempts (timeout 300 docker pull+ backoff) and an honest error: "runner-to-registry connectivity, not this PR. Re-run this job."setup-linux.sh— the get.docker.com convenience script's internal apt/download.docker.com fetches carry no timeout of their own, so a stalled connection hung silently behind the spinner (the fetch of the script itself already hadretry+curl_securebounds). The run is now bounded at 10 minutes (healthy installs take 1–3) and fails with a clear stalled-download message telling the operator to re-run — the installer resumes safely. Same shape as the existing dpkg-lock (#210) and kubectl-fetch bounds;timeoutis coreutils, present on every distro this branch serves.Test plan
timeout 600bound on the get.docker.com branch — written|| return 1-hardened so it stays green under test(bats): make every assertion enforce — 1240 were advisory #527's incoming bats-hygiene gate.setup-linux.bats0 failures;actionlintclean;scripts/gen-manifest.shregenerated (setup-linux.sh is manifested).Prereqs/PATH persistmatrix on this very PR exercises the new pre-pull path live.Refs the 2026-08-04 incident runs (job ids 92054431018, 92073231942).
🤖 Generated with Claude Code
Note
Low Risk
Changes are timeout/retry wrappers around existing CI and installer paths; no auth, data, or cluster logic changes.
Overview
Addresses CI jobs that sat at Installing Docker… or hit the 20-minute timeout when Docker Hub or get.docker.com stalled—infra failures, not product regressions.
CI (
installer-tests.yaml) —distro-prereqsandpath-persistnow pre-pull each matrix image with three bounded attempts (timeout 90 docker pull, backoff) beforedocker run, so implicit pulls can’t hang the job; failures surface as a clear “runner-to-registry connectivity” message.Linux installer (
setup-linux.sh) — The get.docker.com convenience script run is wrapped inspin_cmd_bounded(600s) instead of unboundedspin_cmd, with distinct errors for timeout (124) vs real install failure and prepare-host-aware re-run guidance.Tests / supply chain — Bats pins the 600s bound;
manifest.sha256updated for the lib change.Reviewed by Cursor Bugbot for commit 9b3d81e. Bugbot is set up for automated code reviews on this repo. Configure here.