Uh oh!
There was an error while loading. Please reload this page.
release-train: develop -> staging - #698
Merged
Merged
Conversation
…690) The help text advertised a "868-test bats suite". The real number on develop is 946, and nothing anywhere enforces the two agree — so it drifts on every PR that adds a test and had been wrong for a long time. Re-hardcoding today's value only resets the drift clock. BATS_TEST_COUNT is derived from the source of truth instead: bats declares one test per `@test` at line start, so a grep over scripts/tests/*.bats matches its own count exactly. Verified against a real run — 946 derived, 946 reported. Recursively expanded (`=`, not `:=`) so only `help` pays for the grep. Confirmed by pointing the variable at a marker-touching shell: `make check` — the pre-push path, budgeted under 60 s — never expands it; `make help` does. The same stale 868 appeared a second time in the `check` rationale comment. Prose can't be derived, so the count is simply dropped there; the sentence is about the two-minute runtime, which is the part that actually justifies keeping bats out of `check`. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…(backend#1528) (#692) `resolve-ingestor-digest.sh --write` resolves the `channelTags.prod` float and knew nothing about the ordering ceiling that float sits above. While `serviceDbAccountsByEnv.prod` is false, prod still authenticates as the shared `edgeuser`, so the ingestor it runs must be a release that still HAS the edgeuser fallback. data-ingestors#468 removed that fallback. values.yaml pins `prodDigest` DELIBERATELY behind the float and explains why in prose — but it also tells you to refresh the pin "with the helper, never by hand", and the helper happily resolved straight past the ceiling. client#490 nearly shipped exactly that. Today the float resolves to a digest different from the pin, so the hazard is live, not theoretical. Fail closed in the helper instead of relying on prose: - Refuse `--write` unless `serviceDbAccountsByEnv.prod` is a definite `true`. Absent or unparseable reads also refuse — a chart edit must not be able to silently disarm the guard. - Refuse BEFORE the registry round-trip, so the reason isn't buried under network output and no call is wasted. - `INGESTOR_PIN_ALLOW_PRE_FLAG=1` overrides for a verified target release. - The guard reads the live flag rather than hardcoding a version, so it stops firing on its own once prod flips. Read-only resolution is untouched. Tests: scripts/tests/ingestor-pin-ceiling.bats — refusal, both escape hatches, pin left intact, no-registry-contact ordering, override, post-flip, read-only, sibling-`prod:`-key scoping, and both fail-closed reads. Removing the guard turns 7 of the 10 red.
…ect a false schema doc (#695) * fix(chart): close the CLIENT_ENV + channelTags vocabularies, and correct a false schema doc Three findings from a mutable-tags sweep, all in the same place: the chart documented a vocabulary it never enforced, and in one case documented a fallback it does not have. Follow-up to backend#1723, which fixed the resolution and left the validation open. 1. env.CLIENT_ENV had no `enum`. tracebloc.clientEnv normalized three aliases and passed anything else through RAW. `CLIENT_ENV: prd` rendered jobs-manager:prd, pods-monitor:prd and resource-monitor:prd — tags no producer publishes — missed images.ingestor.channelTags, missed serviceDbAccountsByEnv, AND silently dropped the prod digest pin, which applies only where the env resolves to exactly "prod". Load-bearing in four places, validated in none. The only validator was client-runtime jobs_manager.py's sys.exit(1) on "Unknown CLIENT_ENV" — inside the container that cannot start. Now closed by an `enum` (the primary gate) plus a `fail` in tracebloc.clientEnv (the backstop, for --skip-schema-validation and any repackaging without the schema). 2. channelTags accepted arbitrary keys. `channelTags.staging: 0.7` on a staging edge validated fine and was then ignored — CLIENT_ENV=staging normalizes to stg and the lookup reads channelTags.stg — while `channelTags.stg: 0.7` took effect. The same word, normalized in one place and meaningless one key over. `additionalProperties: false`. 3. The mysql-client tag description said "Empty falls back to env.CLIENT_ENV". The template is `| default "prod"`, and dev/stg/prod all render :prod. tracebloc/mysql-client publishes 8.0, 8.4, four 8.4-<sha> builds, latest and prod — no dev, no stg — so a reader who believed it and set an env-derived tag would pull a nonexistent image AND disarm the mysql-format-guard, which reads "unknown" for an unrecognized tag. Note values.yaml already stated this correctly; the schema — the copy Helm shows customers — held the false half. Description corrected; the template is right and is left alone. BREAKING for any edge deploying an out-of-vocabulary CLIENT_ENV. Such an edge is already broken — it is pulling tags that do not exist — but it now fails at `helm upgrade` rather than at pod start. See the PR body. Tests: the gates cannot be asserted from helm-unittest, which treats a schema violation as a plugin-level error rather than a template failure and offers no way to skip validation and reach the `fail`. So they are exercised from scripts/tests/chart-env-vocabulary.sh (28 checks, every rejection paired with an accept control on the same command line), wired into `make check` and the helm-ci lint job. Five helm-unittest cases pin the mysql-client behaviour the corrected description now describes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(chart): spell the whitespace-only CLIENT_ENV case out instead of hiding it in a word list Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): close 3 Bugbot gaps in the vocabulary test wiring (backend#1723) - chart-env-vocabulary.sh: capture helm --help then match, so grep -q cannot SIGPIPE helm under pipefail and misread the flag as absent (the capture-then-match rule already used for the usermod help probe). - helm-ci.yaml: add scripts/tests/chart-env-vocabulary.sh to the push + pull_request paths filters, so a PR touching only that script still runs the gate it owns. - installer-tests.yaml: add the script to the static shellcheck set (error + warning), re-syncing CI with the Makefile SHELLCHECK_FILES. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* sec(ci): actor-gate the k3s-cuda publish dispatch (backend#1857) build-k3s-cuda.yaml is dispatch-only and pushes ghcr.io/tracebloc/k3s-cuda, and until now nothing authorized the dispatcher. That breaches the decided rule (backend#1422, 2026-08-05): no un-gated workflow_dispatch that publishes. It was missed because it was added one day AFTER the 2026-08-05 inventory was measured, and the client repo had no rows in PUBLISH-PATHS.md at all -- the org's most customer-visible publish path. Found by release-train's publish-inventory-check.sh, which now compares the inventory against every workflow in the fleet. The gate is the fleet's canonical block, copied not reinvented from data-ingestors/release-image.yml, backend/docker-build.yml and client-runtime/publish-images.yml: - a failing STEP, not a skipped job: red and auditable rather than a green run that hides the denial (backend#1424) - BOTH github.actor and github.triggering_actor must pass, because actor stays the ORIGINAL dispatcher on a re-run while triggering_actor is whoever clicked it -- checking one lets a non-allowlisted user replay an allowlisted dispatch (backend#1536) - gated on `inputs.push`, so build-only validation stays open to everyone; the rule is about publishing, not about building Exercised every path against the exact shell the step runs: actor=LukasWodka trigger=LukasWodka ALLOWED actor=saadqbal trigger=saadqbal ALLOWED actor=<other> trigger=<other> DENIED actor=LukasWodka trigger=<other> DENIED <- the replay actor=<other> trigger=LukasWodka DENIED actor=<empty> trigger=<empty> DENIED NOT fixed here, and worth a separate decision: this workflow has NO ref restriction. Its only `if:` was `inputs.push`, so a dispatch from ANY branch could publish, with the image tag built from two user-supplied inputs -- and that constructed tag is what the GPU installer pins. The allowlist reduces this from "any writer" to "two admins"; confining publishes to a reviewed ref (as client-runtime/build-mysql-client.yml does with `github.ref == 'refs/heads/develop'`) is the natural companion and is a policy call, not a copy of an existing pattern. make check: green. actionlint: clean. manifest.sha256: up to date. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * sec(ci): publish k3s-cuda only from a train-owned branch (backend#1857) The actor allowlist bounded WHO could publish. This bounds WHAT can be published, which is the control that actually matters here. install-k8s.ps1:672 defaults $K3S_CUDA_IMAGE to ghcr.io/tracebloc/k3s-cuda:$K8S_VERSION-cuda-$CUDA_BASE_TAG, and its own comment states: "The installer PULLS this image automatically at cluster-create -- the user never builds or pulls anything by hand." So this tag is fetched onto customer machines. Before this change the only `if:` on the job was `inputs.push`, so an allowlisted dispatcher could publish it from ANY branch -- putting unreviewed content behind a customer-pulled tag. staging AND main, not develop: * both are train-owned and protected. develop is the least-reviewed integration branch and has no business behind a customer-pulled tag. * staging is included deliberately, to avoid an ordering trap rather than to be lenient. The tag encodes K8S_VERSION + CUDA_TAG, and check-facts.sh pins those across install-k8s.ps1, the Dockerfile, build.sh and this workflow -- so a bump lands in all four at once. Publishing from staging lets the image for a new tag exist BEFORE main's installer starts asking for it. main-only would leave a window where prod installs pull a tag nobody has pushed yet. Build-only validation is untouched: push=false runs from any branch, by anyone, and the error message says so. The rule is about publishing. Both checks live in ONE step under one `inputs.push` condition, so the gate cannot drift out of sync with the thing it guards. Full matrix exercised against the exact shell: LukasWodka / LukasWodka / refs/heads/main ALLOWED LukasWodka / LukasWodka / refs/heads/staging ALLOWED saadqbal / saadqbal / refs/heads/staging ALLOWED LukasWodka / LukasWodka / refs/heads/develop DENIED LukasWodka / LukasWodka / refs/heads/feat/anything DENIED LukasWodka / LukasWodka / refs/pull/1/merge DENIED <other> / <other> / refs/heads/main DENIED LukasWodka / <other> / refs/heads/main DENIED <- replay NOT closed by this, and filed as backend#1867: the tag stays mutable and the installer pins it by TAG, not digest. Two admins on a protected branch is a process control standing in for a technical guarantee. The org has solved this twice already -- prodDigest for the ingestor, and RFC-BACKEND-1246 for training images -- and k3s-cuda is the remaining customer-pulled image that is neither digest-pinned nor immutable-tagged. make check: green. actionlint: clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
… fails (#672) (#689) * fix(chart): stop init-writable-data skipping the chmod when the chown fails (#672) init-writable-data ran `chown 1000:1000 "$d" && chmod "$m" "$d" || echo …`, so a refused chown short-circuited the chmod and the mode was never applied — while the message said "leaving as-is", implying nothing could be done. That inverts the priority. kubelet ignores fsGroup on hostPath (kubernetes/kubernetes#138411), so the MODE is what makes these trees usable: /data/shared must be other-writable for the ingestion Job (uid 65534, or HOST_UID) and the CLI staging/teardown pod (uid 65532), neither of which is 1000 nor shares a group with it. The chown is cosmetic next to that, and it is also the call most likely to be refused — on a Windows/Docker-Desktop bind mount or an NFS root_squash export it is precisely what fails. So the failure that mattered least was cancelling the one that mattered most, silently nullifying the 2777/3777 split from #667 on the platform that split was written for. Symptom: #653's `mkdir: can't create directory '/data/shared/.tracebloc-staging/': Permission denied`. The chown and the chmod are now separate best-effort statements, each recording whether it failed, and the per-dir verdict is graded on the mode OBSERVED afterwards via `ls -ldn` rather than on either exit status — a bind mount can accept a chmod and ignore it, so an exit code is not evidence. A partial result is reported as such ("chown failed; mode applied anyway") instead of implied. Unchanged: per-dir modes, per-dir independence, non-fatal behaviour, POSIX sh for busybox. Kept diffable by eye against the installer's Get-ReleaseDirsPrepCommand, which already does it this way. Verified by executing the helm-rendered command[2], not by reading it: - sh -n, dash -n, bash --posix -n all clean - busybox:1.35 as root: /data/shared drwxrwsrwx, /data/logs drwxrwsrwt, exit 0 - busybox:1.35 with --cap-drop CHOWN (chown refused, chmod permitted): modes STILL land drwxrwsrwx / drwxrwsrwt; the old command leaves both at drwxr-xr-x - /data/shared read-only (both calls fail): FAIL reported with the real errno, /data/logs still fixed, exit 0 - end-to-end on a shared volume after a refused chown: uid 65534 creates .tracebloc-staging and writes /data/logs; uid 65532 unlinks uid 65534's entries in /data/shared (no sticky) but not in /data/logs (sticky) — both splits intact Tests: the new #672 case fails against the old command and passes against the fix. The obvious comment-scoped guard (`^[^#\n]*chown.*&&.*chmod`) is silently VACUOUS — `${e#*:}` puts a '#' before the chown — so the guard is unscoped and the template describes the old shape in words instead. Existing assertions kept, updated for the multi-line command. jobs_manager_test.yaml 34 -> 35 passing; full suite 379 -> 380 passing with develop's 5 failed / 5 errored baseline unchanged. Also adds the recurring-finding rule to .cursor/BUGBOT.md per CLAUDE.md. Refs #672, #667, #653, #654 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): report only what init-writable-data actually observed (#672) Two overclaims in the first commit's own reporting, both found by running the failure paths rather than reading them — the same family as the bug being fixed. 1. The verdict grades other-writability alone (correctly: that is what decides whether uid 65534/65532 can work, and failing a setgid-stripped-but-writable mount would cry wolf on a working install). But it labelled that bare "OK", which reads as "the whole mode landed". Now says "OK <dir> other-writable" and always prints want vs got, so a mount that granted other-write while dropping S_ISGID is visible instead of implied. 2. Worse: the partial-result note said "mode applied anyway" whenever any call failed. On a dir that was ALREADY other-writable and where BOTH calls were refused, that is simply false — nothing this container did applied anything. Reproduced in busybox:1.35 (pre-set 1777, run as a non-owner uid so chown and chmod are both refused): want 2777 got drwxrwxrwt uid 0 (chown+chmod failed; mode applied anyway) Now reads "(chown+chmod failed; other-writable regardless)" — it claims the observation, not a causal link it cannot support. Re-verified on the helm-rendered command[2]: sh -n / dash -n / bash --posix -n clean; root happy path lands drwxrwsrwx + drwxrwsrwt; chown-refused still lands both modes; already-1777 with both calls refused now reports truthfully; read-only /data/shared still FAILs with the real errno while /data/logs is still fixed; exit 0 throughout. Tests pin both strings, including a notMatchRegex on the old "mode applied anyway" wording. 35 passing, full suite 380 with develop's 5 failed / 5 errored baseline unchanged. Refs #672 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(chart): move the verdict rationale out of the container command (#672) The long "why other-writability alone is the pass condition" prose was inside the script passed to `sh -c`, so it shipped in the pod spec and showed up in every `kubectl get deploy -o yaml`. It belongs in the YAML comment above, which does not. Left a two-line pointer where a script editor will see it. Also records the one intentional divergence from the installer's Get-ReleaseDirsPrepCommand: the chart does not redirect chown/chmod stderr to /dev/null, so the real errno (Operation not permitted vs Read-only file system) lands in `kubectl logs` next to the verdict. The installer suppresses it because its output is a user-facing progress line; an init container's log is a debugging surface, and hiding the errno there would remove the evidence a reader needs. Comment-only inside command[2]: re-rendered and re-ran the chown-refused path in busybox:1.35 to confirm byte-identical output and modes (drwxrwsrwx / drwxrwsrwt, exit 0). 35 passing; full suite 380 passing, baseline unchanged. Refs #672 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(bugbot): correct how long the chained chown/chmod actually shipped (#672) The rule said "for two releases". Verified against git instead: the `chown … && chmod …` shape entered in chart 1.9.20 (#611/#612, commit a07f76b) and survived every version through 1.9.33 — thirteen chart versions, not two. #667 (7852f02) rewrote the modes on that exact line and left the chain untouched, which is the more useful half of the lesson: the line was re-read for its modes and not for its control flow. Also corrects the issue's attribution of the chain to #667. Refs #672, #667, #611 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
LukasWodka
commented
Aug 13, 2026
ContributorAuthor
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 9e3f50d. Configure here.
…prod's (#694) tracebloc.ingestorTag's last-resort literal was a bare "0.8" for every environment. On the `--reuse-values` replay it exists for — a release predating `channelTags`, where no chart default is adopted — a dev or staging edge therefore spawned the PROD ingestor line. That inverts backend#1360: dev/stg channels exist so an ingestor change can be validated on a real edge without a prod release, and an edge silently validating prod's image reports on the wrong artifact. The literal is also no longer merely wrong. The prod float has moved past the ordering ceiling documented at values.yaml `prodDigest`: the 0.8 line no longer carries the ingestor's `edgeuser` DB_USER default that data-ingestors#468 removed (backend#1853). serviceDbAccountsByEnv supplies DB_USER on dev/stg so those two survive it, but that coupling is accidental, and the same literal is where an out-of-vocabulary CLIENT_ENV lands — there serviceDbAccountsByEnv misses too and nothing supplies DB_USER. That is backend#1752 reconstructed from a typo. Keyed on the RESOLVED environment, so the documented aliases reach it too. Also documents the duplication in both directions: the prod literal is a second copy of values.yaml `channelTags.prod` and cannot read the first (a values lookup is nil on exactly the releases this branch serves), so each now points at the other and the suite pins both. An existing case asserted `0.8` for a DEV edge with channelTags absent — the bug written down as an expectation. Replaced by four cases plus a prod control, so "dev falls back to dev" cannot pass by echoing the environment. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…ght the crash-loop (backend#1779) (#699) * fix(chart): give jobs-manager the readiness probe that would have caught the crash-loop (backend#1779) jobs-manager had no probes at all, so the kubelet called the container Ready the moment it was running. That is why `kubectl rollout status` printed "successfully rolled out" over a pod with 4 restarts in 72 seconds on 2026-08-11, and why the E2E agent's client-health step went green (backend#1723, backend#1756). Adds startupProbe + readinessProbe (tcpSocket 8080) to the api container, gated on INGESTION_HTTP_DISABLED with the same truthiness the runtime uses, so an operator who disables the ingestion server does not get a pod that never becomes Ready. Deliberately NO livenessProbe, and the reason is recorded in the template: jobs_manager.py catches a failed run_server_in_thread on purpose ("must not take down the rest of jobs-manager — SB polling can still operate independently"), and a liveness probe on 8080 would reverse that decision from the chart. Readiness removes the pod from the Service, which is the part that needs to happen. A port probe rather than GET /healthz, on purpose: jobs-manager opens 8080 only after backend auth succeeds and the MySQL migration has run, so accepting a connection already implies both. #1779 argues probing /healthz would be a new false-green; for the incident it describes that is not so — the process exited before opening the port, so any probe on 8080 would have been red. See the PR for the evidence. /healthz being an unconditional 200 is still true, and still worth fixing for the narrower "listening but a dependency died" case. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): drop the startupProbe — it kills the container too (Bugbot) Bugbot, Medium, and correct: the startupProbe reversed the very decision this PR argued for when it omitted livenessProbe. A startupProbe is not the gentle cousin of a liveness probe. The kubelet KILLS the container when it fails, exactly as liveness does. So on the one failure mode that motivated backend#1779 -- run_server_in_thread raising, which jobs_manager.py:3535-3542 catches ON PURPOSE because "Startup failure here is fatal for the ingestion flow but must not take down the rest of jobs-manager -- SB polling can still operate independently and training submissions keep working" -- the 30 x 5s budget would expire and CrashLoopBackOff a pod that was still training models. The PR wrote several paragraphs rejecting that outcome for livenessProbe and then shipped it via a different key. readinessProbe alone does the part that needed doing: take the pod out of the Service so nothing routes POST /internal/submit-ingestion-run at a dead port, and stop `kubectl rollout status` reporting success over a crash-loop. It never restarts anything, so the runtime's decision to survive a failed server start stays the runtime's to make. Tests: the startupProbe values test is replaced by one asserting its ABSENCE, with the reasoning above, so this cannot be reintroduced quietly. The disabled-state test no longer asserts notExists on startupProbe -- it does not exist in either state, so that assertion would have passed without saying anything about the gate. 396/396 helm-unittest across 30 suites, make check green, gen-manifest --check clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
LukasWodka
commented
Aug 13, 2026
ContributorAuthor
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 06ed783. Configure here.
…d the Windows prep (#673) (#700) _ensure_release_dirs applied a flat, recursive `chmod -R 777` to data and logs, while Get-ReleaseDirsPrepCommand (#654) and the chart's init-writable-data (#667) both apply a per-dir 2777/3777 split without recursing. Three implementations of one intent, two agreeing and one not — and the odd one out was the copy #667 said should be diffable by eye. Nothing was user-visibly broken: 777 is other-writable and carries no sticky bit, so cross-uid `data delete` worked on the bash path, and on Linux the chart's init container rewrote both dirs at pod start anyway. That absence of a symptom is why the divergence survived two PRs, and why this lands with a test rather than just a fix. - data -> 2777 (setgid, NO sticky: `data delete` unlinks as another uid, #667) - logs -> 3777 (setgid + sticky: nothing has to delete another writer's logs) - drop -R: the dir's own mode governs creation and unlink; recursing stamped setgid/sticky onto every data FILE and walked the whole dataset tree to do it - mysql keeps its recursive 777 — one writer, its own init container, datadir permissions are the database's business (out of scope in #654 for the same reason) - split the pairs on the LAST colon, so a HOST_DATA_DIR containing one can't silently chmod a path that does not exist Tests: hostpath-prep.bats now extracts path:mode pairs from all three sources (bash _release_dirs_spec, the ps1's Get-ReleaseDirsSpec rows + $TB_*_DIR_MODE constants, the chart's init-writable-data loop) and fails if any pair disagrees; cluster.bats asserts the applied modes, that a pre-existing file under data/logs keeps its mode, that mysql stays recursive, and the colon case. Each guard was mutation-checked in all three sources. Closes#673 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
LukasWodka
commented
Aug 13, 2026
ContributorAuthor
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 091e052. Configure here.
LukasWodka
commented
Aug 13, 2026
ContributorAuthor
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 091e052. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated promotion by the release train (RFC-0008 D14). Head is the train-managed
release-train/to-stagingbranch (a mirror ofdevelop), so it never collides with a human PR. Merged only when the fr-gate is green.Note
Medium Risk
Multiple customer-facing chart and hostPath permission behaviors change at upgrade time, though changes are fail-closed, heavily tested, and readiness avoids restart loops on jobs-manager.
Overview
Helm chart 1.9.38 tightens edge configuration and rollout behavior, with matching installer and CI coverage.
env.CLIENT_ENVandimages.ingestor.channelTagsare now closed vocabularies — schemaenum/additionalProperties: falseplus afailintracebloc.clientEnvso typos fail athelm upgradeinstead of pulling wrong tags or dropping the prod digest pin.tracebloc.ingestorTaguses per-env last-resort fallbacks (dev/stg/prod) whenchannelTagsis missing under--reuse-values, instead of always falling back to the prod float. Newscripts/tests/chart-env-vocabulary.shruns inmake check, helm-ci, and installer shellcheck.init-writable-datano longer chainschown && chmod(#672): chmod always runs, success is judged withls -ldn(other-writable), and logs per-dir OK/FAIL without blocking jobs-manager. jobs-manager gets a readiness-only TCP probe on 8080 when ingestion HTTP is enabled (#1779), explicitly without liveness/startup so a degraded ingestion server does not restart training/SB polling.Bash installer (
cluster.sh) applies the same 2777/3777 split as Windows/chart for releasedata/logsdirs (non-recursivechmod).resolve-ingestor-digest.sh --writerefuses to refresh the prod pin whileserviceDbAccountsByEnv.prodis nottrue(ordering ceiling / backend#1528).build-k3s-cudaadds an allowlisted-dispatcher + staging/main ref gate before GHCR publish whenpush=true. Docs/schema fixes for mysql-client empty tag → literal:prod. Chart bumps 1.9.34 → 1.9.38 with expanded unittest/bats and updatedmanifest.sha256forcluster.sh.Reviewed by Cursor Bugbot for commit 091e052. Bugbot is set up for automated code reviews on this repo. Configure here.