Uh oh!
There was an error while loading. Please reload this page.
release-train: staging -> main - #716
Merged
Merged
Conversation
…690) The help text advertised a "868-test bats suite". The real number on develop is 946, and nothing anywhere enforces the two agree — so it drifts on every PR that adds a test and had been wrong for a long time. Re-hardcoding today's value only resets the drift clock. BATS_TEST_COUNT is derived from the source of truth instead: bats declares one test per `@test` at line start, so a grep over scripts/tests/*.bats matches its own count exactly. Verified against a real run — 946 derived, 946 reported. Recursively expanded (`=`, not `:=`) so only `help` pays for the grep. Confirmed by pointing the variable at a marker-touching shell: `make check` — the pre-push path, budgeted under 60 s — never expands it; `make help` does. The same stale 868 appeared a second time in the `check` rationale comment. Prose can't be derived, so the count is simply dropped there; the sentence is about the two-minute runtime, which is the part that actually justifies keeping bats out of `check`. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…(backend#1528) (#692) `resolve-ingestor-digest.sh --write` resolves the `channelTags.prod` float and knew nothing about the ordering ceiling that float sits above. While `serviceDbAccountsByEnv.prod` is false, prod still authenticates as the shared `edgeuser`, so the ingestor it runs must be a release that still HAS the edgeuser fallback. data-ingestors#468 removed that fallback. values.yaml pins `prodDigest` DELIBERATELY behind the float and explains why in prose — but it also tells you to refresh the pin "with the helper, never by hand", and the helper happily resolved straight past the ceiling. client#490 nearly shipped exactly that. Today the float resolves to a digest different from the pin, so the hazard is live, not theoretical. Fail closed in the helper instead of relying on prose: - Refuse `--write` unless `serviceDbAccountsByEnv.prod` is a definite `true`. Absent or unparseable reads also refuse — a chart edit must not be able to silently disarm the guard. - Refuse BEFORE the registry round-trip, so the reason isn't buried under network output and no call is wasted. - `INGESTOR_PIN_ALLOW_PRE_FLAG=1` overrides for a verified target release. - The guard reads the live flag rather than hardcoding a version, so it stops firing on its own once prod flips. Read-only resolution is untouched. Tests: scripts/tests/ingestor-pin-ceiling.bats — refusal, both escape hatches, pin left intact, no-registry-contact ordering, override, post-flip, read-only, sibling-`prod:`-key scoping, and both fail-closed reads. Removing the guard turns 7 of the 10 red.
…ect a false schema doc (#695) * fix(chart): close the CLIENT_ENV + channelTags vocabularies, and correct a false schema doc Three findings from a mutable-tags sweep, all in the same place: the chart documented a vocabulary it never enforced, and in one case documented a fallback it does not have. Follow-up to backend#1723, which fixed the resolution and left the validation open. 1. env.CLIENT_ENV had no `enum`. tracebloc.clientEnv normalized three aliases and passed anything else through RAW. `CLIENT_ENV: prd` rendered jobs-manager:prd, pods-monitor:prd and resource-monitor:prd — tags no producer publishes — missed images.ingestor.channelTags, missed serviceDbAccountsByEnv, AND silently dropped the prod digest pin, which applies only where the env resolves to exactly "prod". Load-bearing in four places, validated in none. The only validator was client-runtime jobs_manager.py's sys.exit(1) on "Unknown CLIENT_ENV" — inside the container that cannot start. Now closed by an `enum` (the primary gate) plus a `fail` in tracebloc.clientEnv (the backstop, for --skip-schema-validation and any repackaging without the schema). 2. channelTags accepted arbitrary keys. `channelTags.staging: 0.7` on a staging edge validated fine and was then ignored — CLIENT_ENV=staging normalizes to stg and the lookup reads channelTags.stg — while `channelTags.stg: 0.7` took effect. The same word, normalized in one place and meaningless one key over. `additionalProperties: false`. 3. The mysql-client tag description said "Empty falls back to env.CLIENT_ENV". The template is `| default "prod"`, and dev/stg/prod all render :prod. tracebloc/mysql-client publishes 8.0, 8.4, four 8.4-<sha> builds, latest and prod — no dev, no stg — so a reader who believed it and set an env-derived tag would pull a nonexistent image AND disarm the mysql-format-guard, which reads "unknown" for an unrecognized tag. Note values.yaml already stated this correctly; the schema — the copy Helm shows customers — held the false half. Description corrected; the template is right and is left alone. BREAKING for any edge deploying an out-of-vocabulary CLIENT_ENV. Such an edge is already broken — it is pulling tags that do not exist — but it now fails at `helm upgrade` rather than at pod start. See the PR body. Tests: the gates cannot be asserted from helm-unittest, which treats a schema violation as a plugin-level error rather than a template failure and offers no way to skip validation and reach the `fail`. So they are exercised from scripts/tests/chart-env-vocabulary.sh (28 checks, every rejection paired with an accept control on the same command line), wired into `make check` and the helm-ci lint job. Five helm-unittest cases pin the mysql-client behaviour the corrected description now describes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(chart): spell the whitespace-only CLIENT_ENV case out instead of hiding it in a word list Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): close 3 Bugbot gaps in the vocabulary test wiring (backend#1723) - chart-env-vocabulary.sh: capture helm --help then match, so grep -q cannot SIGPIPE helm under pipefail and misread the flag as absent (the capture-then-match rule already used for the usermod help probe). - helm-ci.yaml: add scripts/tests/chart-env-vocabulary.sh to the push + pull_request paths filters, so a PR touching only that script still runs the gate it owns. - installer-tests.yaml: add the script to the static shellcheck set (error + warning), re-syncing CI with the Makefile SHELLCHECK_FILES. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* sec(ci): actor-gate the k3s-cuda publish dispatch (backend#1857) build-k3s-cuda.yaml is dispatch-only and pushes ghcr.io/tracebloc/k3s-cuda, and until now nothing authorized the dispatcher. That breaches the decided rule (backend#1422, 2026-08-05): no un-gated workflow_dispatch that publishes. It was missed because it was added one day AFTER the 2026-08-05 inventory was measured, and the client repo had no rows in PUBLISH-PATHS.md at all -- the org's most customer-visible publish path. Found by release-train's publish-inventory-check.sh, which now compares the inventory against every workflow in the fleet. The gate is the fleet's canonical block, copied not reinvented from data-ingestors/release-image.yml, backend/docker-build.yml and client-runtime/publish-images.yml: - a failing STEP, not a skipped job: red and auditable rather than a green run that hides the denial (backend#1424) - BOTH github.actor and github.triggering_actor must pass, because actor stays the ORIGINAL dispatcher on a re-run while triggering_actor is whoever clicked it -- checking one lets a non-allowlisted user replay an allowlisted dispatch (backend#1536) - gated on `inputs.push`, so build-only validation stays open to everyone; the rule is about publishing, not about building Exercised every path against the exact shell the step runs: actor=LukasWodka trigger=LukasWodka ALLOWED actor=saadqbal trigger=saadqbal ALLOWED actor=<other> trigger=<other> DENIED actor=LukasWodka trigger=<other> DENIED <- the replay actor=<other> trigger=LukasWodka DENIED actor=<empty> trigger=<empty> DENIED NOT fixed here, and worth a separate decision: this workflow has NO ref restriction. Its only `if:` was `inputs.push`, so a dispatch from ANY branch could publish, with the image tag built from two user-supplied inputs -- and that constructed tag is what the GPU installer pins. The allowlist reduces this from "any writer" to "two admins"; confining publishes to a reviewed ref (as client-runtime/build-mysql-client.yml does with `github.ref == 'refs/heads/develop'`) is the natural companion and is a policy call, not a copy of an existing pattern. make check: green. actionlint: clean. manifest.sha256: up to date. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * sec(ci): publish k3s-cuda only from a train-owned branch (backend#1857) The actor allowlist bounded WHO could publish. This bounds WHAT can be published, which is the control that actually matters here. install-k8s.ps1:672 defaults $K3S_CUDA_IMAGE to ghcr.io/tracebloc/k3s-cuda:$K8S_VERSION-cuda-$CUDA_BASE_TAG, and its own comment states: "The installer PULLS this image automatically at cluster-create -- the user never builds or pulls anything by hand." So this tag is fetched onto customer machines. Before this change the only `if:` on the job was `inputs.push`, so an allowlisted dispatcher could publish it from ANY branch -- putting unreviewed content behind a customer-pulled tag. staging AND main, not develop: * both are train-owned and protected. develop is the least-reviewed integration branch and has no business behind a customer-pulled tag. * staging is included deliberately, to avoid an ordering trap rather than to be lenient. The tag encodes K8S_VERSION + CUDA_TAG, and check-facts.sh pins those across install-k8s.ps1, the Dockerfile, build.sh and this workflow -- so a bump lands in all four at once. Publishing from staging lets the image for a new tag exist BEFORE main's installer starts asking for it. main-only would leave a window where prod installs pull a tag nobody has pushed yet. Build-only validation is untouched: push=false runs from any branch, by anyone, and the error message says so. The rule is about publishing. Both checks live in ONE step under one `inputs.push` condition, so the gate cannot drift out of sync with the thing it guards. Full matrix exercised against the exact shell: LukasWodka / LukasWodka / refs/heads/main ALLOWED LukasWodka / LukasWodka / refs/heads/staging ALLOWED saadqbal / saadqbal / refs/heads/staging ALLOWED LukasWodka / LukasWodka / refs/heads/develop DENIED LukasWodka / LukasWodka / refs/heads/feat/anything DENIED LukasWodka / LukasWodka / refs/pull/1/merge DENIED <other> / <other> / refs/heads/main DENIED LukasWodka / <other> / refs/heads/main DENIED <- replay NOT closed by this, and filed as backend#1867: the tag stays mutable and the installer pins it by TAG, not digest. Two admins on a protected branch is a process control standing in for a technical guarantee. The org has solved this twice already -- prodDigest for the ingestor, and RFC-BACKEND-1246 for training images -- and k3s-cuda is the remaining customer-pulled image that is neither digest-pinned nor immutable-tagged. make check: green. actionlint: clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
… fails (#672) (#689) * fix(chart): stop init-writable-data skipping the chmod when the chown fails (#672) init-writable-data ran `chown 1000:1000 "$d" && chmod "$m" "$d" || echo …`, so a refused chown short-circuited the chmod and the mode was never applied — while the message said "leaving as-is", implying nothing could be done. That inverts the priority. kubelet ignores fsGroup on hostPath (kubernetes/kubernetes#138411), so the MODE is what makes these trees usable: /data/shared must be other-writable for the ingestion Job (uid 65534, or HOST_UID) and the CLI staging/teardown pod (uid 65532), neither of which is 1000 nor shares a group with it. The chown is cosmetic next to that, and it is also the call most likely to be refused — on a Windows/Docker-Desktop bind mount or an NFS root_squash export it is precisely what fails. So the failure that mattered least was cancelling the one that mattered most, silently nullifying the 2777/3777 split from #667 on the platform that split was written for. Symptom: #653's `mkdir: can't create directory '/data/shared/.tracebloc-staging/': Permission denied`. The chown and the chmod are now separate best-effort statements, each recording whether it failed, and the per-dir verdict is graded on the mode OBSERVED afterwards via `ls -ldn` rather than on either exit status — a bind mount can accept a chmod and ignore it, so an exit code is not evidence. A partial result is reported as such ("chown failed; mode applied anyway") instead of implied. Unchanged: per-dir modes, per-dir independence, non-fatal behaviour, POSIX sh for busybox. Kept diffable by eye against the installer's Get-ReleaseDirsPrepCommand, which already does it this way. Verified by executing the helm-rendered command[2], not by reading it: - sh -n, dash -n, bash --posix -n all clean - busybox:1.35 as root: /data/shared drwxrwsrwx, /data/logs drwxrwsrwt, exit 0 - busybox:1.35 with --cap-drop CHOWN (chown refused, chmod permitted): modes STILL land drwxrwsrwx / drwxrwsrwt; the old command leaves both at drwxr-xr-x - /data/shared read-only (both calls fail): FAIL reported with the real errno, /data/logs still fixed, exit 0 - end-to-end on a shared volume after a refused chown: uid 65534 creates .tracebloc-staging and writes /data/logs; uid 65532 unlinks uid 65534's entries in /data/shared (no sticky) but not in /data/logs (sticky) — both splits intact Tests: the new #672 case fails against the old command and passes against the fix. The obvious comment-scoped guard (`^[^#\n]*chown.*&&.*chmod`) is silently VACUOUS — `${e#*:}` puts a '#' before the chown — so the guard is unscoped and the template describes the old shape in words instead. Existing assertions kept, updated for the multi-line command. jobs_manager_test.yaml 34 -> 35 passing; full suite 379 -> 380 passing with develop's 5 failed / 5 errored baseline unchanged. Also adds the recurring-finding rule to .cursor/BUGBOT.md per CLAUDE.md. Refs #672, #667, #653, #654 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): report only what init-writable-data actually observed (#672) Two overclaims in the first commit's own reporting, both found by running the failure paths rather than reading them — the same family as the bug being fixed. 1. The verdict grades other-writability alone (correctly: that is what decides whether uid 65534/65532 can work, and failing a setgid-stripped-but-writable mount would cry wolf on a working install). But it labelled that bare "OK", which reads as "the whole mode landed". Now says "OK <dir> other-writable" and always prints want vs got, so a mount that granted other-write while dropping S_ISGID is visible instead of implied. 2. Worse: the partial-result note said "mode applied anyway" whenever any call failed. On a dir that was ALREADY other-writable and where BOTH calls were refused, that is simply false — nothing this container did applied anything. Reproduced in busybox:1.35 (pre-set 1777, run as a non-owner uid so chown and chmod are both refused): want 2777 got drwxrwxrwt uid 0 (chown+chmod failed; mode applied anyway) Now reads "(chown+chmod failed; other-writable regardless)" — it claims the observation, not a causal link it cannot support. Re-verified on the helm-rendered command[2]: sh -n / dash -n / bash --posix -n clean; root happy path lands drwxrwsrwx + drwxrwsrwt; chown-refused still lands both modes; already-1777 with both calls refused now reports truthfully; read-only /data/shared still FAILs with the real errno while /data/logs is still fixed; exit 0 throughout. Tests pin both strings, including a notMatchRegex on the old "mode applied anyway" wording. 35 passing, full suite 380 with develop's 5 failed / 5 errored baseline unchanged. Refs #672 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(chart): move the verdict rationale out of the container command (#672) The long "why other-writability alone is the pass condition" prose was inside the script passed to `sh -c`, so it shipped in the pod spec and showed up in every `kubectl get deploy -o yaml`. It belongs in the YAML comment above, which does not. Left a two-line pointer where a script editor will see it. Also records the one intentional divergence from the installer's Get-ReleaseDirsPrepCommand: the chart does not redirect chown/chmod stderr to /dev/null, so the real errno (Operation not permitted vs Read-only file system) lands in `kubectl logs` next to the verdict. The installer suppresses it because its output is a user-facing progress line; an init container's log is a debugging surface, and hiding the errno there would remove the evidence a reader needs. Comment-only inside command[2]: re-rendered and re-ran the chown-refused path in busybox:1.35 to confirm byte-identical output and modes (drwxrwsrwx / drwxrwsrwt, exit 0). 35 passing; full suite 380 passing, baseline unchanged. Refs #672 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(bugbot): correct how long the chained chown/chmod actually shipped (#672) The rule said "for two releases". Verified against git instead: the `chown … && chmod …` shape entered in chart 1.9.20 (#611/#612, commit a07f76b) and survived every version through 1.9.33 — thirteen chart versions, not two. #667 (7852f02) rewrote the modes on that exact line and left the chain untouched, which is the more useful half of the lesson: the line was re-read for its modes and not for its control flow. Also corrects the issue's attribution of the chain to #667. Refs #672, #667, #611 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…prod's (#694) tracebloc.ingestorTag's last-resort literal was a bare "0.8" for every environment. On the `--reuse-values` replay it exists for — a release predating `channelTags`, where no chart default is adopted — a dev or staging edge therefore spawned the PROD ingestor line. That inverts backend#1360: dev/stg channels exist so an ingestor change can be validated on a real edge without a prod release, and an edge silently validating prod's image reports on the wrong artifact. The literal is also no longer merely wrong. The prod float has moved past the ordering ceiling documented at values.yaml `prodDigest`: the 0.8 line no longer carries the ingestor's `edgeuser` DB_USER default that data-ingestors#468 removed (backend#1853). serviceDbAccountsByEnv supplies DB_USER on dev/stg so those two survive it, but that coupling is accidental, and the same literal is where an out-of-vocabulary CLIENT_ENV lands — there serviceDbAccountsByEnv misses too and nothing supplies DB_USER. That is backend#1752 reconstructed from a typo. Keyed on the RESOLVED environment, so the documented aliases reach it too. Also documents the duplication in both directions: the prod literal is a second copy of values.yaml `channelTags.prod` and cannot read the first (a values lookup is nil on exactly the releases this branch serves), so each now points at the other and the suite pins both. An existing case asserted `0.8` for a DEV edge with channelTags absent — the bug written down as an expectation. Replaced by four cases plus a prod control, so "dev falls back to dev" cannot pass by echoing the environment. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…ght the crash-loop (backend#1779) (#699) * fix(chart): give jobs-manager the readiness probe that would have caught the crash-loop (backend#1779) jobs-manager had no probes at all, so the kubelet called the container Ready the moment it was running. That is why `kubectl rollout status` printed "successfully rolled out" over a pod with 4 restarts in 72 seconds on 2026-08-11, and why the E2E agent's client-health step went green (backend#1723, backend#1756). Adds startupProbe + readinessProbe (tcpSocket 8080) to the api container, gated on INGESTION_HTTP_DISABLED with the same truthiness the runtime uses, so an operator who disables the ingestion server does not get a pod that never becomes Ready. Deliberately NO livenessProbe, and the reason is recorded in the template: jobs_manager.py catches a failed run_server_in_thread on purpose ("must not take down the rest of jobs-manager — SB polling can still operate independently"), and a liveness probe on 8080 would reverse that decision from the chart. Readiness removes the pod from the Service, which is the part that needs to happen. A port probe rather than GET /healthz, on purpose: jobs-manager opens 8080 only after backend auth succeeds and the MySQL migration has run, so accepting a connection already implies both. #1779 argues probing /healthz would be a new false-green; for the incident it describes that is not so — the process exited before opening the port, so any probe on 8080 would have been red. See the PR for the evidence. /healthz being an unconditional 200 is still true, and still worth fixing for the narrower "listening but a dependency died" case. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): drop the startupProbe — it kills the container too (Bugbot) Bugbot, Medium, and correct: the startupProbe reversed the very decision this PR argued for when it omitted livenessProbe. A startupProbe is not the gentle cousin of a liveness probe. The kubelet KILLS the container when it fails, exactly as liveness does. So on the one failure mode that motivated backend#1779 -- run_server_in_thread raising, which jobs_manager.py:3535-3542 catches ON PURPOSE because "Startup failure here is fatal for the ingestion flow but must not take down the rest of jobs-manager -- SB polling can still operate independently and training submissions keep working" -- the 30 x 5s budget would expire and CrashLoopBackOff a pod that was still training models. The PR wrote several paragraphs rejecting that outcome for livenessProbe and then shipped it via a different key. readinessProbe alone does the part that needed doing: take the pod out of the Service so nothing routes POST /internal/submit-ingestion-run at a dead port, and stop `kubectl rollout status` reporting success over a crash-loop. It never restarts anything, so the runtime's decision to survive a failed server start stays the runtime's to make. Tests: the startupProbe values test is replaced by one asserting its ABSENCE, with the reasoning above, so this cannot be reintroduced quietly. The disabled-state test no longer asserts notExists on startupProbe -- it does not exist in either state, so that assertion would have passed without saying anything about the gate. 396/396 helm-unittest across 30 suites, make check green, gen-manifest --check clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…d the Windows prep (#673) (#700) _ensure_release_dirs applied a flat, recursive `chmod -R 777` to data and logs, while Get-ReleaseDirsPrepCommand (#654) and the chart's init-writable-data (#667) both apply a per-dir 2777/3777 split without recursing. Three implementations of one intent, two agreeing and one not — and the odd one out was the copy #667 said should be diffable by eye. Nothing was user-visibly broken: 777 is other-writable and carries no sticky bit, so cross-uid `data delete` worked on the bash path, and on Linux the chart's init container rewrote both dirs at pod start anyway. That absence of a symptom is why the divergence survived two PRs, and why this lands with a test rather than just a fix. - data -> 2777 (setgid, NO sticky: `data delete` unlinks as another uid, #667) - logs -> 3777 (setgid + sticky: nothing has to delete another writer's logs) - drop -R: the dir's own mode governs creation and unlink; recursing stamped setgid/sticky onto every data FILE and walked the whole dataset tree to do it - mysql keeps its recursive 777 — one writer, its own init container, datadir permissions are the database's business (out of scope in #654 for the same reason) - split the pairs on the LAST colon, so a HOST_DATA_DIR containing one can't silently chmod a path that does not exist Tests: hostpath-prep.bats now extracts path:mode pairs from all three sources (bash _release_dirs_spec, the ps1's Get-ReleaseDirsSpec rows + $TB_*_DIR_MODE constants, the chart's init-writable-data loop) and fails if any pair disagrees; cluster.bats asserts the applied modes, that a pre-existing file under data/logs keeps its mode, that mysql stays recursive, and the colon case. Each guard was mutation-checked in all three sources. Closes#673 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
#701) * sec(#1876): grant jobs-manager list nodes for GPU->CPU pending fallback The check_pending_jobs GPU->CPU fallback (client-runtime#217) calls list_node() to tell an autoscaling-in GPU node from a genuinely absent one. nodes is cluster-scoped, so the grant can only live in the clusterScope: true ClusterRole (like tokenreviews). Without it list_node() 403s and the runtime fail-safe assumes a GPU node is present, silently disabling the fallback — a Pending GPU pod wedges forever while the backend shows RUNNING. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(chart): bump client 1.9.38 → 1.9.39 for the list-nodes RBAC grant The rbac.yaml ClusterRole change is packaged chart content, so it only reaches installs via a new chart version (chart-version-guard). Merge current develop in and bump version + appVersion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ckend#1853) (#697) * fix(ci): watch every mutable label that points at a pinned digest (backend#1853) On 2026-08-12 the ingestor's `channelTags.prod: "0.8"` float moved to v0.8.8, which removed the legacy edgeuser DB_USER fallback. Nothing noticed; a manual sweep found it. Default prod edges were spared only because prodDigest pins v0.8.2 and prodPin defaults to true. DESIGNED TO SAADQBAL'S DIAGNOSIS, not to the symptom: "The disease is a moving label pointing at an immutable trust decision, with no watcher on the label. Anything about the build could have been the thing that changed -- the fallback just got there first." So this script asserts NOTHING about a build's contents. There is no DB_USER check, deliberately: such a check goes green the next time something else moves, which is the failure this watches for rather than a variant of it. It asks one property-agnostic question per pin -- does the float still resolve to the digest we decided to trust? The trusted versions are registered where they already were: the digest:/ prodDigest: fields of client/values.yaml. No second list to keep in sync; adding a pin enrols it automatically. Against the real chart: 3 pins found, squid agrees, the ingestor DRIFTS (the real finding), and mysqlClient is reported UNWATCHABLE -- it carries a pin but declares no repository, so nothing can tell you when its trust decision goes stale. That is a genuine modelling gap, reported rather than skipped. TWO BUGS OF MY OWN, both found by running it, not reading it: 1. images:-scoped discovery watched 1 of the 3 pins. squid's pin lives OUTSIDE images:, and an "empty field means skip" rule dropped mysqlClient without a word. Rewritten pin-driven over the whole file: a pin is watched or REPORTED, never skipped. 2. IFS=$'\t' collapses runs of tabs, because tab is IFS whitespace. A record with an empty repository AND tag slid the pin into the wrong variable, leaving $pin empty, and the row was skipped in silence. This is the exact defect release-train's own parse-repos suite pins by name; 0x1f is not whitespace. Also in this commit, per review: * client/values.yaml no longer states the float's version. It said "at v0.8.4" while the float was at v0.8.8 -- and any version written there is stale on the next release, reading as reassurance ("two patches behind") for a gap that may be far larger. The comment now says the float moves without us and points at this watcher. * docs/SECURITY.md 4.1.1 is reframed as an explicit CEILING with a table: v0.8.0-v0.8.4 safe with the flag off, v0.8.8+ NOT (config.py read at each tag). It previously implied the unsafe release was hypothetical; it exists and is what the float points at. Tests: 16 bats cases, registry stubbed via a documented seam that prints STUBBED on every run so a log cannot pass as a real audit. The repo's bats-hygiene guard caught that 31 of my assertions were advisory -- a bare [ ] on a non-final line cannot fail its test -- so all are now || return 1. Mutation-verified after that fix: comparison always true -> 4 tests fail discovery restricted to the images: block -> test 6 fails a pin with no repository silently skipped -> tests 9 + 11 fail (An earlier sed-based mutation of the third reported 0 failures; the pattern contained backticks and never matched. Inert mutation, not coverage -- re-done with an asserted anchor.) NOT in `make check`: needs network + docker, and is knowingly red today (the drift IS the finding). Runs daily via digest-drift.yml; the bats suite is in `make bats`, which needs neither. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(ci): bound the digest-drift registry lookups + bump chart version (backend#1853) - resolve_index_digest ran docker buildx imagetools inspect / docker manifest inspect with no timeout, so a wedged daemon or stuck registry hung the daily job instead of failing closed as UNRESOLVED. Wrap them in _tmout (timeout/gtimeout, 30s) so a stuck call is non-zero -> UNRESOLVED. - client/values.yaml changed, so the chart-content gate requires a Chart.yaml version bump: 1.9.34 -> 1.9.35. * fix(chart): match appVersion to version (Bugbot, #1853) The version bump left appVersion at 1.9.34; app.kubernetes.io/version follows appVersion, so installed objects would advertise the old chart version. Keep them in lockstep as this chart does: appVersion 1.9.35. * fix(ci): discover single-quoted and off-structure digest pins, fix checkout label check-digest-drift.sh only matched a double-quoted sha256 at exactly four-space indent, so a single-quoted (digest: SQ...SQ) or more-deeply nested pin was dropped in silence -- and the PINS==0 guard cannot catch that while any one conforming pin remains, so a run could print 'no drift' with a pin unwatched. Make discovery quote- and indent-agnostic: a canonical pin is watched as before; a pin off the structure is REPORTED unwatchable, never skipped. Adds bats cases for the single-quoted repro, the good-masks-sneaky case, and the off-structure -> UNWATCHABLE path. Also correct digest-drift.yml's checkout pin comment: 11d5960a is v4.4.0 (repo-standard, on releases/v4), not v5.0.0; drop the stray double space. addresses @saadqbal review, Bugbot, client#697. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…on Always (#569) (#705) * fix(chart): re-image control-plane pods by digest instead of relying on Always (client#569) The always-running control-plane pods rendered `imagePullPolicy: Always`, so a Docker Desktop / WSL2 restart forced a registry round-trip and landed in ImagePullBackOff even with the image already cached in containerd — the edge came back only once docker.io was reachable, up to ~6h behind Docker Hub's anonymous pull-rate limit. `Always` could not simply be flipped: it IS the update mechanism. The image-refresh CronJob's `kubectl rollout restart` only picks up a new build because the pull policy re-resolves the floating tag. Offline-safety and restart-driven updates are mutually exclusive unless the image REFERENCE changes on update. So the reference is now what changes. - All four control-plane call sites render IfNotPresent unconditionally: jobs-manager (api + pods-monitor sidecar), requests-proxy, resource-monitor. - image-refresh swaps `rollout restart` for `kubectl set image repo@digest`. - Two workloads come under refresh for the first time, both quietly broken before: requests-proxy runs the SAME jobs-manager image but was never reconciled (it skewed until an unrelated restart, then jumped to whatever the tag pointed at), and resource-monitor had no deliberate update path at all. - requests-proxy follows the jobs-manager digest in the same tick, no second registry HEAD. images.requestsProxy.digest opts it out. - resource-monitor needs a second Role in the node-agents namespace: get/patch resourceNames-scoped to the one DaemonSet, list/watch namespace-wide and read-only (RBAC ignores resourceNames for collection verbs, and `rollout status` requires them). Not rendered when resourceMonitor: false. - imageRefreshEnabled now requires all three refreshed images pinned before retiring the CronJob; pinning only the two class-1 images used to render it away, leaving the DaemonSet with no update path. - Private mirrors: the script resolves digests from docker.io, so pinning one onto a mirrored reference could pin an image the mirror does not hold. It logs and goes inert. Under `rollout restart` that mismatch was merely useless; with `set image` it has to fail closed. The first-tick "record without acting" contract is kept deliberately: re-imaging on the first tick would rewrite repo:tag to repo@digest for byte-identical content on every fresh install, rolling the Deployment and the DaemonSet on every node for nothing. A fresh edge therefore runs repo:tag until the first real digest change — restart-safe offline, just not yet reproducible. Two bounded limitations are documented in the script header rather than hidden: a chart version bump re-renders repo:tag and this tick will not re-pin (the annotation still matches), and skew predating this change is prevented but not repaired. Both self-heal at the next upstream release and neither can break a running edge. The proper fix for both is reconciling against each workload's live container image instead of a shared annotation — which `set image` makes possible for the first time, and which is a deliberate follow-up. Verified: helm unittest 405 passed, failures exactly develop's pre-existing baseline (5 failed / 5 errored, diffed against a stashed baseline run — zero new). helm lint clean, all four client/ci value sets render, check-style and check-facts pass, gen-manifest --check up to date. Chart 1.9.38 -> 1.9.39, version + appVersion in lockstep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): close two Bugbot findings on the digest-on-update reconcile (client#569) 1. Jobs-manager pin skipped requests-proxy (_helpers.tpl:219). requests-proxy runs the jobs-manager IMAGE, and both the helper and the values docs claimed it follows the jobs-manager pin — but the Deployment read only `images.requestsProxy.digest`. Pinning jobs-manager ALONE was therefore a silent trap: the proxy kept rendering the floating `repo:tag` while jobs-manager ran the pinned digest, AND image-refresh skips pinned images, so nothing ever wrote a `set image` for the proxy either. It froze on the tag indefinitely, running a different build of the same image — precisely the skew #569 exists to close, re-introduced through the pinning path. `images.requestsProxy.digest` now falls back to `images.jobsManager.digest` when empty, so the claim is true by construction. The proxy key remains an explicit per-workload override. The helper comment now records that the test depends on that fallback, so removing it forces requests-proxy back into the "nothing left to do" test. 2. Refresh budget too small for three rollouts (image-refresh-cronjob.yaml). A tick now waits on up to three sequential `rollout status` calls, each up to rolloutTimeout (10m), while activeDeadlineSeconds was a hardcoded 1800 — exactly 3 x 10m, zero headroom. Blowing it is not a benign timeout: the Job is killed mid-wait, so under `set -e` the post-success annotate never runs, the recorded digest never advances, and the shared `refresh-attempt` counter stays incremented. Three such ticks trip the #563 flap lockout for EVERY control-plane image at once, while the CronJob still looks healthy. - activeDeadlineSeconds is now `imageRefresh.activeDeadlineSeconds`, default 3600 (3 x the default rolloutTimeout plus 100% slack), with a schema entry (minimum 60) and the raise-both-together constraint documented on rolloutTimeout. - The resource-monitor DaemonSet gets an explicit `updateStrategy.rollingUpdate.maxUnavailable: 10%`. Kubernetes defaults to maxUnavailable: 1, so a digest change converged in (nodes x pull+start) and blew the 10m wait on any multi-node cluster long before the image was bad. Safe to widen here specifically: resource-monitor is a read-only node metrics reader, so a briefly absent pod degrades scheduling telemetry and nothing else. Kubernetes rounds 10% down and floors it at 1, so small clusters keep today's one-at-a-time behaviour. Verified: helm unittest 411 passed, failures still exactly develop's pre-existing baseline (5 failed / 5 errored, diffed against a stashed baseline run — zero new). helm lint clean, check-style clean. Rendering confirms requests-proxy resolves to the jobs-manager digest when only jobsManager is pinned, activeDeadlineSeconds renders 3600 and honours an override, the DaemonSet carries maxUnavailable 10%, and the schema rejects a sub-60 deadline. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): keep Always where the digest reconcile cannot run (client#569) Bugbot, High severity — a regression introduced by this PR's first commit. Making `imagePullPolicy: IfNotPresent` UNCONDITIONAL removed the only update path from every edge where the replacement mechanism cannot run. `Always` on a floating tag is not merely offline-fragility; it IS an update path — restart the pod and the kubelet re-resolves the tag. Two configurations relied on exactly that and were left frozen on their cached image forever, with a green CronJob and no signal: * `global.imageRegistry` (private mirror). The reconcile resolves digests from docker.io, so this PR deliberately makes it inert there rather than pin a digest the mirror may not hold. With IfNotPresent on top, syncing the mirror and restarting kept serving the cached tag. * `imageRefresh.enabled: false`. values.schema.json has always promised these operators the image stays put "until manual restart" — true only with Always. The inert-path log message this PR added made it worse by telling operators to "sync your mirror and restart the workloads", which under IfNotPresent does nothing. That guidance is now correct because the policy is correct. The policy is now resolved per image by one helper, `tracebloc.controlPlanePullPolicy`, so the four call sites cannot disagree: 1. explicit `digest` pin -> IfNotPresent (immutable reference; updates come from changing the pin) 2. reconcile can run here -> IfNotPresent (`set image` changes the REFERENCE, which is what the kubelet pulls) i.e. the CronJob renders AND images come from docker.io 3. neither -> Always (floating tag + restart is the only update path that edge has) The trade is deliberate: offline-restart safety is delivered precisely where the digest reconcile can deliver updates. An edge that opts out of the mechanism keeps pre-#569 semantics rather than silently freezing — a frozen control plane with no signal is worse than a restart that needs the network. Verified by rendering the full matrix: default and all-pinned resolve IfNotPresent across jobs-manager (both containers), requests-proxy and resource-monitor; mirror and refresh-disabled resolve Always across all of them; a pin plus refresh-disabled correctly splits (pinned container IfNotPresent, unpinned sibling Always). helm unittest 417 passed, failures still exactly develop's pre-existing baseline (5 failed / 5 errored — zero new). helm lint, check-style clean; all four client/ci value sets render. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(chart): correct statements #569 made false in values.yaml and the schema (client#569) Self-audit follow-through, same class as the Bugbot findings on this PR: behaviour changed and the prose describing it did not. Per CLAUDE.md, a change that makes a statement false fixes that statement in the same PR. Docker Hub rate-limit arithmetic (the operationally significant one). #569 raised the per-tick cost from 2 manifest HEADs to 3 — resource-monitor joined jobs-manager and pods-monitor; requests-proxy adds none because it reuses the jobs-manager digest. values.yaml still claimed "2 images x 4/hr = 8/hr, well under the cap". It is now 12/hr, so 72 per 6h against the anonymous cap of 100, and the headroom left for OTHER workloads sharing the egress IP fell from ~52 to ~28. That is worth more than an arithmetic fix: exhausting this cap behind a shared corporate NAT is one of the failure modes in the incident #569 exists to fix — a rate-limited edge cannot pull for up to ~6h. The comment now states the real numbers, names the shared-IP risk, and gives the concrete remedy (a 3/hr schedule restores ~9/hr, and a digest pin drops that image's HEAD entirely). The default schedule is deliberately UNCHANGED: it is still within the cap, and quietly slowing drift pickup fleet-wide is a call for a human, not a side effect of a doc fix. values.schema.json descriptions, which were describing the pre-#569 mechanism: - imageRefresh: said it "rolls the deployment"; now describes `kubectl set image` across the three workloads, and why that is what permits IfNotPresent. - imageRefresh.enabled: says explicitly that disabling falls back to Always, so the long-standing "until manual restart" promise stays true. - imageRefresh.schedule: three manifests per tick, not two, with the numbers. - imageRefresh.maxRefreshAttempts: re-imaging, not re-restarting; notes the counter is shared across workloads. - images.jobsManager.digest: records that requests-proxy inherits it. - images.requestsProxy.digest: documents the follow-jobs-manager default (it had no description at all), so the pinning trap Bugbot found is discoverable from the schema rather than only from the template comment. - images.podsMonitor / resourceMonitor.digest: descriptions added. Checked and deliberately NOT changed: docs/SECURITY.md 6.1's `rollout restart` after a secret rotation restarts the pod to re-read the Secret, not to pull an image, so it is unaffected by the pull-policy change. Verified: helm unittest 417 passed, failures still exactly develop's baseline (zero new). helm lint and check-style clean; schema is valid JSON. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): one definition for "resource-monitor needs no refresh" (client#569) Bugbot, Medium — and, as it notes, the SAME helper-vs-runtime disagreement class this PR already had to fix for requests-proxy. The rule was written twice and the two copies drifted: * `tracebloc.imageRefreshEnabled` treated resource-monitor as done only when `images.resourceMonitor.digest` was set. * The CronJob's RESOURCE_MONITOR_PINNED env ALSO treated `resourceMonitor: false` as done — correctly, since with no DaemonSet a cross-namespace `set image` would just fail the tick. So `resourceMonitor: false` plus both class-1 images pinned kept rendering a CronJob that skipped every image and exited green every 15 minutes, forever. Before #569 that combination retired the CronJob cleanly. It is also exactly the green-forever-while-doing-nothing failure mode the script's own #571 comment warns about. Both consumers now read one helper, `tracebloc.resourceMonitorRefreshPinned`, which is the single place the "nothing to do for resource-monitor" rule lives: an explicit digest pin, or the DaemonSet disabled outright. Nil-safe, and an absent `resourceMonitor` key reads as enabled to match the `ne .Values.resourceMonitor false` gate on the DaemonSet itself. Three tests: the combination now retires both the CronJob and its RBAC, and the opposite direction is guarded too — disabling resource-monitor must NOT retire the CronJob while jobs-manager or pods-monitor can still drift. Verified: helm unittest 420 passed, failures still exactly develop's baseline (zero new). Rendering confirms image-refresh is gone for the retiring combination (only auto-upgrade's CronJob remains) and present in both keep cases. helm lint, check-style clean; all four client/ci value sets render. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): give the node-agents image-refresh RBAC a distinct name (client#569) Bugbot, High. The node-agents Role and RoleBinding added by #569 reused `tracebloc.imageRefreshName` — the same name as the release-namespace pair. `nodeAgents.namespace.name` pointing back at the release namespace is a SUPPORTED layout, not a misconfiguration: node-agents-namespace.yaml documents it and deliberately skips creating the Namespace in that case. In that layout both pairs land in one namespace, so the chart rendered two Roles and two RoleBindings with identical names in identical namespaces. Confirmed by rendering before the fix: Role tracebloc t-image-refresh x2 RoleBinding tracebloc t-image-refresh x2 Helm then either refuses the release or lets the later DaemonSet-only Role overwrite the deployments Role. The second outcome is the dangerous one: it is silent, and it strips image-refresh's patch on jobs-manager and requests-proxy. Because #569 also moves those pods to IfNotPresent, they would be left with no update path at all — the exact failure this PR exists to prevent, reached through a different door. The Role and RoleBinding now use `tracebloc.imageRefreshNodeAgentsName` (`<release>-image-refresh-node-agents`). The RoleBinding SUBJECT deliberately keeps the un-suffixed name: there is only one ServiceAccount and it lives in the release namespace. A distinct name is correct in BOTH layouts — split namespaces get one Role each, and the collapsed layout gets two complementary Roles (deployments, daemonsets) bound to the same SA, which is the intended grant. Verified by rendering both layouts: no duplicate (kind, namespace, name) in either, subjects and roleRefs resolve to the right objects in both. Tests pin the new names, the subject/roleRef split, and add a collapsed-layout regression case. helm unittest 421 passed, failures still exactly develop's baseline (zero new); helm lint and check-style clean; all four client/ci value sets render. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(chart): collapse the requests-proxy name to one definition, pin the rest (client#569) Proactive follow-up, not a review finding. Three of the five Bugbot findings on this PR were the same mistake: one side of a two-sided contract moved and the other did not (the requests-proxy digest pin, the resource-monitor pin signal, and the node-agents RBAC name). That is one habit, not three coincidences, so I audited the rest of the diff for the same shape and found two more live instances — both in contracts #569 itself created or started depending on. 1. Workload names. #569 made image-refresh reconcile workloads BY NAME with `kubectl set image`, which gives those names a second consumer. A rename that reached only the workload template would leave the CronJob patching something that does not exist: the tick fails, the digest record freezes, and the shared flap counter eventually locks out refresh for every control-plane image. `<release>-requests-proxy` had exactly two call sites — the Deployment, and the CronJob env I added — so it is now one definition, `tracebloc.requestsProxyName`. resource-monitor already went through `tracebloc.resourceMonitorName`. `<release>-jobs-manager` is deliberately NOT unified here. It has six call sites across five files (NOTES.txt, the PDB, tracebloc.serviceAccountName, ...), most of which this change does not otherwise touch, and the repo convention is that refactors ship separately from behaviour changes. Half-migrating it would BE the bug this commit is about. A contract test pins the two sides in the meantime. 2. Container names. `kubectl set image <workload> <container>=<ref>` fails outright on a wrong container name, so `api`, `pods-monitor-container`, `proxy` and `tracebloc-resource-monitor` are a hard contract between the script and the workload templates, previously asserted from neither side. Tests now pin both sides of everything still spelled out twice: the three reconcile target names and all four container names, asserted from the CronJob AND from jobs_manager / requests_proxy / resource_monitor. Renaming either side alone now fails CI instead of silently breaking refresh on the fleet. Verified: rendered names are byte-identical before and after the unification (t-requests-proxy in both the Deployment and REQUESTS_PROXY_DEPLOYMENT), and the requests-proxy Service selector is untouched — it matches the pod label `app: requests-proxy`, not the Deployment name. helm unittest 426 passed, failures still exactly develop's baseline (zero new); helm lint and check-style clean; all four client/ci value sets render. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…al one (#702) * fix(installer): last failing command wins, so the report names the real one The ERR recorder shipped in #683 kept the FIRST failure. That is wrong, and a field report showed why: the run reported Stopped at .../lib/common.sh:527 (exit 1). command: sudo -n true for a failure two steps later. common.sh:527 is _real_sudo, reached from step a's _probe_privilege, whose `sudo -n true` returns non-zero to mean "a password is needed" — the installer then PRINTS that as a normal row in the host check. The trap fires for every failing command, benign ones included, so first-wins latched onto a routine probe inside a step that SUCCEEDED and refused every later record. The fatal command was never captured. A confidently wrong location is worse than the blank screen #683 replaced: it sends the reader to a line that is working as designed. Last-wins is precise. errexit stops the script AT the fatal command, and the trap fires once per failing command with no per-frame re-firing as the error unwinds — verified on bash 3.2 (macOS) and 5.x. Also: - Re-entrancy guard. `set -E` makes the recorder inherit its own trap, and the new `log` call is exactly the kind of command that fails inside it (its `[[ -n "${LOG_FILE:-}" ]] && …` form returns non-zero with no log open). Without the guard that recurses forever. - install_cleanup disarms the ERR trap before reading the record. Its own lines fail routinely — a `kill` on a dead pid, a false `[[ … ]]` — and under last-wins each would overwrite the fatal command with a cleanup detail. - The full ERR trail now goes to the log. The benign entries are not noise: reading them in order is what identified this bug. Five bats tests, mutation-real against the first-wins guard, including the field shape end to end — a probe that fails inside an `if`, a step that then succeeds, a fatal command afterwards. 982 bats green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(common): make the _record_err re-entrancy test actually exercise the guard (client#702) The recursion test drove the ERR trap with `command false || true`, which does not fire it — a command in a || list is excluded from ERR (bash manual). On bash 5.3 that form fires the trap zero times, so _record_err never ran and SURVIVED printed with or without the guard. bash 3.2 does fire it, which is why it looked green on macOS while being vacuous on Linux CI. `unset LOG_FILE` was the other half: log() is `[[ -n $LOG_FILE ]] && echo …`, so with no log open the write never attempts and nothing inside the recorder fails. The failure has to come from the redirection, so point LOG_FILE at a path whose parent does not exist (fails for root too, unlike chmod 000). Fixing only that is not enough. bash re-enters an ERR trap at most once, so deleting _TB_IN_RECORD_ERR does not hang anything and the survival test passes either way. Add a test for what the guard actually protects: a re-entrant call must not overwrite TB_ERR_* with the recorder's own log failure, which would turn 'died at helm upgrade' into 'died writing its log'. Verified by mutation — with the guard removed, the new test goes red and the survival test stays green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Shujaat Hasan <shujaat@tracebloc.io>
release-train: develop -> staging
… an old CLI (#708) Once a machine is "already set up", no installer on any platform ever updated the tracebloc CLI again. The cluster keeps upgrading itself hourly via the auto-upgrade CronJob; the host binary does not, and nothing noticed. A field machine was found on CLI v0.5.1 from 6 July against a current 0.10.6 — five minors behind — on a box whose chart had meanwhile auto-upgraded 1.8.5 -> 1.9.34. The user re-ran the newest installer and it exited before reaching the CLI step. macOS and Linux (one file, no platform branch, so identical): install-k8s.sh:198 assess -> healthy -> hands off -> exit 0 install-k8s.sh:241 install_tracebloc_cli, the ONLY call site, never reached _assess_cli_present checked presence, not version. Note the asymmetry: a MISSING CLI was correctly degraded/cli-missing and got installed; only a STALE one slipped through, the single state nothing checked. Windows had the same hole plus a worse one. Test-ToolsPresent covers docker/kubectl/k3d/helm — the CLI is not in it at all — and `completed` is set purely from ClientState -eq "connected", which says nothing about the CLI, while Install-TraceblocCli is deliberately non-fatal. So a machine whose CLI install FAILED was still marked complete and never retried: permanently CLI-less, not merely stale. The CLI's own nudge cannot rescue either case. It landed in v0.10.0, is nudge-only, needs an interactive TTY, and is skipped under CI / without a config dir — and by definition cannot reach anyone below v0.10.0, which is exactly the population at risk. Both fast paths now check the version, with 0.10.0 as the floor: the release from which the CLI can keep itself current. It is a FLOOR, not a "must be latest", so it costs no network call per run and never needs raising. Below it -> degraded/cli-outdated (bash) / fall through to Install-TraceblocCli (Windows), which upgrades it once. Both fail OPEN on an unreadable version — that is not evidence of staleness, and reinstalling the CLI on every run would be worse than the staleness. _version_lt is self-contained: no `sort -V` (BSD sort predates it) and no jq, and it orders 0.9.9 below 0.10.0 rather than lexically. Either fix alone leaves a platform broken, so they land together. 988 bats + 670 Pester green; the fast-path guards on both sides are mutation-real. Closes#707 Refs tracebloc/backend#1920 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…175 (client#703) (#704) * fix(macos): give install_macos the Tier 0 path Linux has had since #1175 A Mac where a container runtime is already installed AND running is classified Tier 0 — "zero root, no privileged steps" — and step b then ran the admin gate and primed sudo anyway, demanding an administrator password to install a runtime that was already installed and answering. A field report died exactly there: Host check Container runtime Docker 29.7.2 — docker info OK ✓ → Install tier Tier 0 (zero root) — a container is already runnable. ... step b: install_macos starting (OS=Darwin ARCH=arm64 tier=0) step b: admin check passed <- nothing after this install_linux has short-circuited Tier 0 since RFC 0001 #1175; install_macos was a flat sequence with no tier branch at all, and install_macos_cli_tools hardcoded /usr/local/bin + sudo under the comment "macOS has no Tier/rootless model" — so even skipping the admin gate would still have prompted for a password to write the tools. Tier 0 now skips the admin gate, sudo priming, Homebrew and the Docker Desktop install, and lands the pinned tools in ~/.local/bin with no sudo (the same target _set_tools_target picks on Linux), then persists it on PATH via _persist_tools_on_path — which self-gates on that directory and is already macOS-aware. It still verifies amd64 emulation and still sets up login autostart, both of which are genuinely needed and neither of which wants admin. Skipping the admin gate is deliberate, not incidental: Tier 0 is precisely the case RFC 0001 opened up — a user with NO administrator rights on a machine where someone else already provisioned the runtime. That user could not install at all before this. Every other tier is byte-identical, including an UNSET INSTALL_TIER (a stale bootstrap that never fetched probe.sh), which keeps the privileged path. Seven bats tests; the four that matter are mutation-real — removing either the install_macos branch or the cli-tools target branch fails them. 985 bats green. Closes#703 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(macos): Tier 0 autostart makes no sudo call (Bugbot, client#704) The Tier 0 path prints "no administrator rights needed", then called _install_macos_autostart, whose headless branch writes a system LaunchDaemon via sudo mkdir / sudo tee / sudo launchctl. On a headless Mac (SSH with a TTY) that prompts for a password — the exact step-b failure Tier 0 exists to remove — and a Ctrl-C at the prompt leaves the install half-done before cluster create. _install_macos_autostart now takes an optional "no-sudo" argument. In that mode the GUI LaunchAgent path is unchanged (it never needed sudo and is the correct macOS analogue of a user-level autostart), but the headless LaunchDaemon path — the only branch that needs root — is skipped with instructions on how to enable reboot autostart later, instead of calling sudo. The Tier 0 caller passes "no-sudo"; the privileged path is byte-identical and still installs the daemon. This mirrors Linux, whose Tier 0 already stays out of privileged autostart. Four new mutation-real bats tests (24 green): headless no-sudo makes zero sudo calls and writes no daemon; GUI no-sudo still installs the user LaunchAgent; the Tier 0 caller passes no-sudo; the privileged caller does not. shellcheck clean at error severity; scripts/manifest.sha256 regenerated (R8). Bugbot, client#704. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(macos): headless Tier 0 names colima in the reboot footer, not Docker Desktop (Bugbot, client#704) Headless Tier 0 skips the boot LaunchDaemon rather than prompt for the password Tier 0 exists to avoid, so TB_MACOS_AUTOSTART stays unset. But _reboot_note's not-configured branch is the macOS/Windows GUI fallback: it printed 'open Docker Desktop to bring tracebloc back'. On a headless Mac that names a runtime which is not what runs here, an action there is no desktop to perform, and it directly contradicts the 'colima start' hint the skip printed moments earlier. That footer is the LAST line of a successful install, so it is the advice the operator actually leaves with. Set TB_MACOS_HEADLESS_NO_AUTOSTART at the skip site and give _reboot_note a branch for it. A configured autostart still wins, so a stale marker cannot downgrade a real promise. Tests cover both summary branches, that the skip path actually sets the marker (otherwise the new branch is unreachable and the tests are theatre), and that a GUI session does not. The existing golden 'open Docker Desktop' assertion is unchanged. manifest.sha256 regenerated; make check green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(macos): resolve the headless recovery command, don't assume colima (Bugbot + @saadqbal, client#704) My previous commit fixed the footer naming Docker Desktop on a headless box, but replaced it with an unconditional 'colima start' — the same defect in the other direction, and inconsistent with the privileged path ten lines below, which already resolves `command -v colima` and softens to a generic message when it is absent (#430 Bugbot). Three cases now: * boot daemon ALREADY installed — Tier 0 means someone else provisioned the box, so a prior admin install may have left it. Skipping the write is still right (we hold no sudo), but claiming manual recovery would be false. Set TB_MACOS_AUTOSTART and return 0. * colima resolvable — name 'colima start', as before. * colima absent — say 'start your Docker runtime manually' and name nothing. The summary reads the command the skip site resolved against this host rather than hardcoding one, so the two can't disagree. Tests for all three, plus the generic summary branch. The no-colima test overrides `command -v` for colima alone rather than emptying PATH, which would also remove date(1) via log and test the harness instead of the branch. 31/31 lifecycle, 28/28 summary, hygiene 18/18, make check green, manifest regenerated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Shujaat Hasan <shujaat@tracebloc.io>
…lient pin watchable (#714) * fix(chart): un-vacuum the order-of-ops assertion, and make the mysqlClient pin watchable Two Bugbot Mediums on the staging promotion PR #713. 1. ORDER-OF-OPS ASSERTION WAS VACUOUS. It required `kubectl rollout restart` before `kubectl annotate` -- but #569 REMOVED that command, and a sibling notMatchRegex forbids it on a command line. So the only text it could still match was the header comments explaining the old mechanism, and it passed regardless of what order the real commands ran in. Measured on the rendered script rather than argued: with the post-re-image annotates hoisted above the first `kubectl set image` -- the actual regress, header comments left in place -- the OLD assertion still passes (True) and the NEW one fails (False). On the real script the new one passes. So a regress that recorded the digest before re-imaging would have frozen every workload on its old image with a green suite. Now anchored to COMMAND lines and to the mechanism that exists: `set image` -> `rollout status` -> `annotate`. (Two earlier attempts at that mutation were themselves broken -- one hoisted the annotates above the comments too, one popped by stale indices after inserting. Both "proved" the wrong thing. The mutation above is verified to have actually moved the lines before its result is quoted.) 2. mysqlClient PIN WAS NEVER WATCHED. `images.mysqlClient` carried a real digest but no `repository` and an empty `tag`, because both live as template defaults. check-digest-drift.sh pairs a pin with the repo/float in its own block, found neither, and classified it UNWATCHABLE -- which exits non-zero, so the DAILY DRIFT WATCH STAYS RED FOREVER while the pin it exists to compare is never checked. A guard that cannot pass teaches everyone to ignore it. Stated in values and consumed by the template, so the value is real config rather than decoration. Both are what the templates already defaulted to. Verified: * render byte-identical across aks/bm/eks values (only POD_TOKEN_SIGNING_SECRET differs, which helm regenerates every run); mysql image line unchanged * helm lint clean; helm unittest 36/36 including the edited suite * drift watch: BEFORE "UNWATCHABLE: images.mysqlClient is pinned to sha256:f546…" AFTER "ok tracebloc/mysql-client:prod sha256:f546e47fb339…" * test(image-refresh): pin the order guard to the digest annotate, not any annotate The order-of-ops assertion only required SOME `kubectl annotate deployment` after set-image/rollout-status. The flap-counter-reset annotate (`${ATTEMPT_KEY}- ${FLAP_KEY}-`) already sits there by design, so moving ONLY the digest-recording annotate (`$annotate_args`) above the re-image kept the suite green while the digest was recorded BEFORE the image was applied — the freeze path this guard exists to catch. Require the `$annotate_args` continuation line so the ordering is tied to the digest write itself, distinct from the flap reset. Verified by mutation: the loose pattern stayed green with the digest annotate moved above set-image; the tightened one fails. Bugbot, client#714. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(chart): bump to 1.9.40 for the chart-content changes in this PR The version-bump-gate (scripts/chart-version-guard.sh) requires a Chart.yaml version bump when client/templates|values change, since the Helm repo only publishes a NEW version — an unbumped edit ships dark or overwrites a published one. develop is at 1.9.39; bump to 1.9.40. backend#1468 unrelated — this is client#714. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…end#1729 sweep 5) (#706) * ci(env): one CLIENT_ENV vocabulary, four languages, now checked (backend#1729 sweep 5) Sweep 5 of the inert-verification epic. The epic states the class as: "A verification written in the same vocabulary as the thing it verifies cannot detect a vocabulary error." and its evidence was 366 chart tests covering dev/stg/prod/unset/unknown while NOT ONE set `staging` -- the alias the chart's own docs recommend. That measured gap is now closed (staging has 9 cases, development 2, production 4). What is NOT closed is the structure underneath it: the same three alias->canonical mappings are declared FOUR times, in four languages, and nothing compares them. 1 client/templates/_helpers.tpl $aliases := dict ... Go template 2 client/values.schema.json the CLIENT_ENV enum JSON Schema 3 scripts/lib/common.sh tb_client_env() bash case 4 scripts/install-k8s.ps1 Get-TraceblocClientEnv PowerShell switch Two of the four have already drifted, separately, and been repaired separately: backend#1723 fixed the chart, backend#1745 fixed the bash installer -- whose own comment records the cost, "a raw `staging` fell through to the prod branch, so verify_credentials() checked staging credentials against the production backend and reported them invalid". So adding a seventh spelling to the template leaves both installers silently not reducing it, which is #1745 reintroduced in a repo that has already paid for it once. THE GUARD DERIVES, IT DOES NOT RESTATE. It parses all four declarations and compares them to each other; it holds no copy of the vocabulary, because a fifth hand-written list is the defect rather than the fix (the lesson of backend#1780 and backend#1828, where hand-copied declarations each claimed the others kept them honest and nothing crossed the boundary). It also asserts every accepted spelling is exercised by at least one helm-unittest case -- the specific thing #1729 measured. ARMED WHILE GREEN, deliberately: all four agree today and all six spellings are tested, so this imports no backlog. Arming a red check trains people to skip the tier -- the same reasoning as .github#235 and the opposite of what happened when a fleet-wide copies: bump reddened the org audit for 2h20m on 2026-08-12. Mutations, all five behaving correctly: template gains a 4th alias, installers unchanged 3 findings bash reducer drops staging (the #1745 shape) 1 finding PowerShell maps staging -> dev 1 finding schema accepts a spelling no reducer maps 2 findings a parser goes stale (dict renamed) EXIT 2, fail-closed That last one matters most: zero parsed pairs compares equal to zero parsed pairs, so a stale parser would report agreement between four declarations it never read. It exits 2 with a diagnostic instead. Wired into SHELLCHECK_FILES and `make check` beside its sibling, and into helm-ci with path filters covering all four declarations -- including scripts/lib/common.sh and scripts/install-k8s.ps1, so a change to either installer runs it. make check green; chart-env-vocabulary 28/28; gen-manifest --check clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(env-check): don't misreport a missing python3 as a closed-vocabulary gap env-vocabulary-agreement.sh reads the CLIENT_ENV enum out of values.schema.json with python3, but `make setup` never installed (or even checked for) python3, so on the pre-push `make check` path a missing interpreter failed closed with the false diagnosis that the schema has no CLIENT_ENV enum and the vocabulary is no longer closed. The Go/bash/PowerShell reducers are jq-free by rule, so python3 is the JSON parser here and was the one unguarded dependency. Preflight `command -v python3` before the enum is read, and at the call site branch on the helper's exit status: 3 (its sys.exit for a genuinely-absent enum) still reports the real closed-vocabulary finding, while any other non-zero (a python3 that fails to run, malformed JSON) reports a distinct tooling/parse error. Add python3 to `make setup`'s prereq check so the gap is caught up front. Bugbot, client#706. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(env): shellcheck env-vocabulary-agreement in the static job The Makefile's SHELLCHECK_FILES already lists env-vocabulary-agreement.sh, but installer-tests.yaml's static job shellchecked only through chart-env-vocabulary.sh, so CI never linted the new guard. Append it to both the error-gate and warning-advisory invocations so the workflow list matches the Makefile again (same files, same order). Bugbot, client#706. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
release-train: develop -> staging
LukasWodka
commented
Aug 14, 2026
ContributorAuthor
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8059981. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated promotion by the release train (RFC-0008 D14). Head is the train-managed
release-train/to-mainbranch (a mirror ofstaging), so it never collides with a human PR. Merged only when the fr-gate is green.Note
High Risk
Changes fleet control-plane image update semantics, offline-restart behavior, and ingestor/env validation on customer clusters; misconfiguration could freeze images or break ingestion DB auth if floats advance past pinned digests.
Overview
Client chart 1.9.40 ships a rework of image-refresh (#569): the CronJob now
kubectl set imagejobs-manager, requests-proxy, and resource-monitor (plus node-agents RBAC) instead ofrollout restart, withtracebloc.controlPlanePullPolicysoIfNotPresentapplies only where digest reconcile or explicit pins provide an update path—Alwaysremains for private mirrors and disabled refresh. Related fixes include requests-proxy followingjobsManager.digest, DaemonSetmaxUnavailable: 10%, and a higheractiveDeadlineSecondsfor three sequential rollouts.Install-time safety tightens
CLIENT_ENV(schemaenum+tracebloc.clientEnvfail) and per-env ingestor fallbacks (dev/stg/prodinstead of prod0.8for all).init-writable-dataruns chown and chmod independently and verifies other-writable modes (#672). jobs-manager gets a TCP readiness probe on :8080 (readiness only—no liveness/startup) when ingestion HTTP is enabled (#1779). Cluster RBAC addsnodeslist for GPU→CPU pending fallback (#1876).CI/supply chain: daily
digest-driftworkflow andcheck-digest-drift.sh, GHCR k3s-cuda publish gated on allowlisted actors and main/staging refs,helm-vocab/ env-vocabulary tests inmake check, and SECURITY.md updates on the 0.8.8 / #468 ingestor boundary.Reviewed by Cursor Bugbot for commit 8059981. Bugbot is set up for automated code reviews on this repo. Configure here.