Uh oh!
There was an error while loading. Please reload this page.
release-train: develop -> staging - #647
Merged
Merged
Conversation
#637) * fix(chart): image-refresh builds explicit in-cluster kubeconfig (#634) kubectl in the alpine/k8s image does not reliably auto-detect in-cluster config in the image-refresh Job pod: with no kubeconfig on disk it falls back to http://localhost:8080 and every API call fails, even though the SA token and KUBERNETES_SERVICE_* env are present. This made the refresh tick fail on every run fleet-wide (dev/stg/prod), surfacing only as the generic "cannot read deployment (NotFound, RBAC denial, or API error)" because the read gate swallowed kubectl's stderr. - Build an explicit in-cluster kubeconfig from the mounted ServiceAccount (server from KUBERNETES_SERVICE_*, ca + token from the projected SA volume) before any API call, so bare `kubectl` targets the real API server. Verified in a dev pod under the real pod constraints (readOnlyRootFilesystem, HOME=/home/kubectl, emptyDir mounts). - Surface the real API error on the fail path instead of >/dev/null 2>&1, so the next incident is one log line, not a live repro session. - Regression guards lock the kubeconfig preamble in place. - Bump chart 1.9.15 -> 1.9.16 (script is chart content; chart-version-guard). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(chart): address Bugbot — tokenFile off argv + IPv6-safe API host (#634) Two Bugbot findings on the in-cluster kubeconfig preamble: - SA bearer token was on the kubectl argv via `--token="$(cat ...)"` (readable through procfs). Use `kubectl config set users.*.tokenFile` — the kubeconfig stores only the PATH, no token value on argv or in the file, and kubectl re-reads the rotating projected token per call. - The server URL concatenated host:port without IPv6 brackets, an invalid URL when KUBERNETES_SERVICE_HOST is IPv6 (client-go uses net.JoinHostPort). Bracket the host for dual-stack / IPv6 clusters. Regression guards updated: assert bracketed api_host + tokenFile, and a notMatchRegex forbidding a regression back to `set-credentials --token`. Verified in a dev pod (readOnlyRootFilesystem, real SA): tokenFile path set, no token value embedded, bare kubectl rollout status + get both rc=0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(#555): add activeDeadlineSeconds to the auto-upgrade Job autoUpgrade.timeout ("10m") bounds only `helm upgrade --wait`, not the earlier `helm repo add`/`helm repo update` HTTPS fetches. On a TCP black-hole (a corporate proxy that drops packets with no RST) those fetches can hang with no hard timeout; with concurrencyPolicy: Forbid the hung Job then makes every future scheduled tick "still active" and skipped, wedging auto-upgrade until a human kills the pod. Add activeDeadlineSeconds (new autoUpgrade.activeDeadlineSeconds knob, default 900s) to the Job spec as a hard wall-clock backstop, mirroring storage-assertions-check.yaml. Kept comfortably above the 600s helm timeout plus repo-update headroom so a slow-but-healthy upgrade is never killed mid-flight. Also declared in values.schema.json alongside the sibling knobs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#555): bump Chart.yaml version and nil-guard activeDeadlineSeconds Bump chart version 1.9.21 -> 1.9.22 so the chart-content change can be published (CI gate), and keep appVersion in lockstep with version ("1.9.22") per the repo convention — the previous divergence was the Bugbot finding. Default autoUpgrade.activeDeadlineSeconds to 900 in the CronJob template so a --reuse-values upgrade that omits the new key still renders a valid integer instead of an empty/invalid value. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#555): never let the Job deadline undercut a longer helm timeout Bugbot: the fixed default activeDeadlineSeconds (900s) could kill a healthy `helm upgrade --wait` when autoUpgrade.timeout was configured longer (e.g. 30m), firing before helm's own timeout. Render activeDeadlineSeconds as max(configured floor, parsed timeout + 300s repo-fetch headroom) via a new tracebloc.durationSeconds helper. The 300s mirrors the baked-in 900-600 default headroom, so the default case is a no-op (900) and the `| default 900` reuse-values nil-guard is preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(#561): run macOS install steps under bounded spinners Homebrew install, `brew install docker/colima`, `colima start`, and the `hdiutil` DMG install ran under spin_cmd, which has no deadline. A wedged brew (network stall), a hung colima start (stale VZ VM), or hdiutil on a bad DMG would hang the install indefinitely behind a spinning cursor with no timeout or kill. Move all five to spin_cmd_bounded with sensible deadlines (Homebrew 30m to allow a fresh-Mac Command Line Tools pull; the others 15m each), matching the create/helm paths that already bound themselves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(installer): regenerate manifest for setup-macos.sh change (#624) setup-macos.sh changed in this PR; scripts/manifest.sha256 was stale, failing the Static analysis --check gate. Regenerated via scripts/gen-manifest.sh; --check now passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ptops (#625) * fix(#562): soften API-wait and readiness timeouts for slow/proxied laptops F8 (_wait_for_api): the hard 60s cap (max=30 x sleep 2) false-failed a slow machine still loading images on its first kubectl even after `k3d --wait` returned. Replace with a seconds-based, env-tunable budget (TB_API_WAIT_S, default raised to 180s), and update the error to note a re-run is safe and how to extend the wait. F9 (READY_TIMEOUT): raise the default 300s -> 600s so several GB of images over a slow/proxied link doesn't leave a healthy-but-slow machine reported as CLIENT_STATE=starting / "not connected". Updated in all three synced consumers (scripts/spec/facts.env, summary.sh, install-k8s.ps1); check-facts.sh passes. Still overridable via the READY_TIMEOUT env var. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#562): align check-facts.bats fixtures with READY_TIMEOUT 600 spec (#625 Bugbot CID 3727520250) The spec (facts.env) raised READY_TIMEOUT 300 -> 600, but the bats fixtures still seeded the summary.sh / install-k8s.ps1 consumers at 300. That made the green-path --check test false-fail (a real drift the gate must NOT report) and left the drift test rewriting 300 -> 600, which no longer exercised drift once the spec was 600. Seed both consumers at 600 to match the spec, push the ps1 to a wrong value (900) in the drift test so the gate genuinely reports ReadyTimeout drift, and bump the --write test spec to 900 so it actually re-stamps both bash and PowerShell budgets. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…626) * fix(#563): back off image-refresh on a flapping-readiness deployment A deployment whose readiness oscillates (becomes Ready, then crashes after warmup right around the rollout-status timeout) fails the acting tick's `rollout status`. Under `set -e` the tick exits before the success annotation lands, so `recorded` never advances to the new digest and every later settled tick re-issues the identical `rollout restart` (~every schedule) — churning ReplicaSets on a genuinely flapping deployment. Record the ATTEMPT (target-digest signature + a consecutive-failure count) in a `tracebloc.io/refresh-attempt` annotation BEFORE the restart, so the counter survives the set -e exit. After imageRefresh.maxRefreshAttempts (default 3) in-a-row failures for the same target, skip the restart and surface the flapping deployment instead of re-restarting forever. The counter resets when the target digest changes or a rollout finally succeeds. New value + schema entry + env wiring; default installs stay byte-identical apart from the new env var. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#563): resume image-refresh once a flapping deployment settles + bound annotate (#626 review) Address PR #626 review: - Backoff was a dead end: after MAX_REFRESH_ATTEMPTS failures the tick exited before restarting forever, so a deployment that later became healthy/settled never got the reconciling rollout and stayed on the old image until a newer digest appeared. Now, while in back-off (which is only reached on a tick that already passed the settled guard), each settled observation advances a cooldown; after BACKOFF_RESUME_AFTER settled ticks the attempt counter resets and refresh retries. A cleared flap resumes and reaches `recorded == latest`; a still-flapping deployment is retried at most once per cooldown, never every tick. - Bound the new pre-restart `kubectl annotate` calls with --request-timeout=15s (repo rule for non-watch kubectl). - values.yaml comment updated to describe the resume-on-settle behaviour. - Chart.yaml version 1.9.15 -> 1.9.16 (chart content changed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#563): read the back-off attempt counter fail-closed (#626 review) The attempt-counter read used `get_annotation ... || true`, and get_annotation piped `kubectl | jq` with no pipefail. A failed kubectl/jq therefore collapsed to an empty value that the back-off logic read as "no prior attempts" (0) -- failing OPEN past the guard and re-restarting a possibly-flapping deployment (violates the no-fail-open-guards rule). get_annotation now captures kubectl separately and returns non-zero on either a kubectl or jq error, so a caller can distinguish a genuinely absent annotation (success + empty output -> 0 attempts) from a read error (non-zero return). The attempt read takes the SAFE branch on a read error: skip the restart this tick rather than churn the deployment. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#563): resume grants one restart, not a burst; fix cooldown off-by-one (#626 review) Two back-off bugs in the image-refresh cooldown logic: 1. Cooldown resume restarted in bursts (CID 3728456918). On resume the counter was reset to 0, handing a still-flapping deployment a fresh budget of MAX_REFRESH_ATTEMPTS restarts across the next consecutive settled ticks -- a ReplicaSet-churn burst that broke the "at most once per cooldown" invariant. Now resume sets prev_count = MAX-1 so exactly ONE restart is issued; if it fails, the persisted count is MAX and the next settled tick re-enters back-off immediately. A healthy deployment still succeeds on that one attempt and clears the counter. 2. Cooldown off-by-one delayed resume (CID 3728456929). The branch compared settled_ticks (excluding the current tick) against BACKOFF_RESUME_AFTER, so the tick logging "Settled 4/4" still incremented and exited, and resume happened only the following tick (~15m late). Now it compares the count INCLUDING this settled tick, so 4/4 resumes on that tick, matching the log and the "after BACKOFF_RESUME_AFTER settled ticks" contract. Both fixes reuse the existing single-counter encoding; no new state added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#563): bound get_annotation read with --request-timeout=15s so a wedged API fails closed, not hangs (#626 Bugbot CID 3728757993) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#563): replace image-refresh back-off with a plain consecutive-failure counter (#626 Bugbot CID 3728806670) Simplify the over-engineered cooldown/auto-resume/signature machinery down to the minimal design #563 calls for. The single-integer annotation is no longer overloaded (signature + count + cooldown offset); it is now a plain count of consecutive failed refresh attempts that means one thing. Removed: - the cooldown counter, BACKOFF_RESUME_AFTER, cooldown_seen/settled-tick counting, and the resume-to-MAX-1 single-restart logic (auto-resume); - the signature built from annotate_args and its target-change reset. This is the fragile "partial resolve resets back-off" path (Bugbot CID 3728806670) -- deleting it eliminates the finding. Kept (the correct #563 core): - settled-guard short-circuit for unsettled deployments; - digest-change detection (refresh only when the resolved digest changed); - fail-closed annotation reads (skip the tick on a kubectl/jq read error, never treat an error as "0 attempts"); - --request-timeout=15s on every non-watch kubectl call. New semantics: increment the counter (persisted before the restart) when a rollout is issued but does not settle; reset to 0 ONLY on a genuinely successful settled rollout; after MAX_REFRESH_ATTEMPTS in a row, STOP restarting, WARN, and annotate tracebloc.io/refresh-flap-detected for a human / monitoring. No auto-resume -- a human clears the counter (or a rollout that finally settles resets it). Net -30 lines. helm lint + template render clean; embedded script passes sh -n / dash -n; tick-by-tick simulation confirms flap->stop+flag (no burst), healthy->reset, read-error->skip, new-digest->refresh, unsettled->short-circuit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#563): make the image-refresh success reset bounded + independent of the digest write (#626 Bugbot CID 3729110045) On a successful settled rollout the consecutive-failure counter (tracebloc.io/refresh-attempt) was cleared only by being batched into the final digest-record annotate, which had no --request-timeout. If that annotate hung or failed under `set -e`, the counter stayed elevated and the digest went unrecorded, so later ticks re-restarted, burned the budget, and could permanently trip the flap lockout while the CronJob stayed green. Clear the counter (and flap marker) in its own --request-timeout=15s annotate BEFORE the digest record, so the success reset is bounded and no longer contingent on the digest write succeeding. Also bound the digest-record annotate and the rollout restart with --request-timeout=15s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ds (#616) (#639) * fix(gpu): correct COPY --exclude placement in the k3s-CUDA Dockerfile (#616) The multi-stage rootfs copy failed with "cannot copy to non-directory: .../bin": --exclude was placed BEFORE the args (`COPY --from=k3s --exclude=/bin / /`), which the labs frontend didn't apply, so /bin (a real dir in the static-k3s image) was copied over Ubuntu's merged-/usr /bin symlink and collided. Match the official k3d recipe: `--exclude=/bin` AFTER the args, then copy /bin explicitly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): correct COPY --exclude (before args, relative path) + verify rootfs (Bugbot) (#639) A TRAILING `--exclude=/bin` is parsed as the COPY DESTINATION, so the k3s rootfs copied to /--exclude=/bin instead of / — the build passed but the image was BROKEN (no rootfs overlay). My earlier "fix" (moving --exclude after the args) hit exactly this. Correct form: --exclude BEFORE src/dest, with the path RELATIVE to the source (`bin`, not `/bin` — the absolute path never matched, which is why the very first build collided on /bin). Also make build.sh self-verifying: build + load, export the image filesystem, and FAIL if it contains a `--exclude` artifact or is missing /bin/k3s — so a mis-parse can never publish a broken image again. Only push after the rootfs verify passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): exclude all merged-/usr dirs so the k3s rootfs overlays cleanly (#639) The before-args `--exclude=bin` fixed /bin but the build then collided on /lib: Ubuntu 22.04 is merged-/usr, so /bin /sbin /lib /lib64 are all symlinks to /usr/*, while the Alpine k3s image ships them as real dirs. k3s is a static binary (no shared-lib deps), so exclude all four (keeping Ubuntu's glibc userland for nvidia-ctk) and overlay only k3s's static /bin into /usr/bin via the kept /bin symlink. build.sh rootfs verify relaxed to accept usr/bin/k3s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(gpu): exclude the full Ubuntu merged-/usr + /var symlink set (#639) /lib was not the last collision — /var/run (symlink to /run) collided next. Exclude every Ubuntu-22.04 top-level symlink dir preemptively (bin/sbin/lib/lib32/lib64/libx32 + var/run, var/lock); excluding a path k3s does not have is a harmless no-op. k3s is static, so keeping Ubuntu's glibc userland under those paths and overlaying only k3s's /bin is sufficient. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…xy Deployment (#641) * fix(#1143): add startup/liveness/readiness probes to requests-proxy The requests-proxy Deployment declared ports/securityContext/resources/env but no probes, so the pod was marked Ready the instant the container process started -- before gunicorn had bound :8888. `tb doctor`'s "Service Bus egress (requests-proxy) ready" check reads ReadyReplicas, so it showed a false green even when the proxy could not relay. Add tcpSocket startup/liveness/readiness probes on the gunicorn listen port (8888), mirroring mysql-deployment.yaml's three-probe structure. tcpSocket (not httpGet) because the proxy application lives in the separate client-runtime repo and exposes no chart-visible HTTP health path; a probe against an unknown path would 404 and wedge the pod, whereas a successful TCP accept is the honest "gunicorn is serving" signal the doctor check needs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(chart): bump version+appVersion to 1.9.26 (#1143) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(#1143): make requests-proxy probe comment honest about tcpSocket tcpSocket:8888 confirms the gunicorn master has bound the listen socket — a strict improvement over no probes that catches a container that never starts — but it does not fully close the false-green: the master binds before the worker finishes importing create_app() and keeps the socket bound while a worker crash-loops, so a proxy that can't relay can still read Ready. A real /healthz in client-runtime would be needed to close it fully. Probes themselves unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…t can't OOM-kill it (#642) * fix(#1144): pin jobs-manager memory requests==limits so mass restart can't OOM it On single-node cluster recovery (Docker/k3d restart -> every pod restarts at once and contends for node CPU + memory) the jobs-manager pod was killed with exit 137, both the `api` and `pods-monitor` containers, taking ~9m to recover. During that churn the client stops heartbeating and `tb` shows the alarming "couldn't confirm it's connected to tracebloc". Root cause: both containers ran Burstable QoS with requests.memory (512Mi/256Mi) well below limits (1Gi/512Mi). Under node memory pressure the kernel OOM-killer prefers the pod using more than its request, so jobs-manager was the victim. Fix (chart resources): pin requests.memory == limits.memory for both containers (api 1Gi, pods-monitor 512Mi) so each holds a Guaranteed memory reservation and is no longer the node's largest OOM victim -- the same requests==limits pattern mysql-deployment.yaml already documents. Also raise requests.cpu (api 100m->250m, pods-monitor 50m->100m) so a mass restart doesn't throttle their startup to the observed ~9m crawl. CPU limits and memory limits are unchanged. Updated both the values.yaml defaults and the template inline fallbacks (used on helm upgrade --reuse-values from a release predating these keys). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(chart): bump version+appVersion to 1.9.27 (#1144) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#1144): reword "Guaranteed QoS" to the accurate oom_score_adj mechanism (#642 review) The pod is Burstable (cpu request != limit); only memory is pinned. Pinning requests.memory==limits.memory lowers the container's oom_score_adj, making it a less-preferred node OOM victim — same pattern mysql uses (mysql also omits its cpu limit). Reworded the templates, values.yaml, and the test description. Also fixed the failing jobs_manager_test resource assertions: api requests.memory 512Mi -> 1Gi, pods-monitor requests.memory 256Mi -> 512Mi. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
#643) * sec(#1528): provision tb_meta + tb_ingest service-account Secrets (S1) Chart side of backend#1528 S1 (RFC-0003 D10 close-out), paired with tracebloc/client-runtime's minting PR. jobs-manager mints two dedicated, single-database MySQL identities to replace the root-equivalent edgeuser; this wires the Secrets and env they read. - values: serviceDbAccounts (default false) + optional tbMetaPassword / tbIngestPassword operator pins, mirroring perExperimentDbCreds / credmgrPassword. Added to values.schema.json. - secrets.yaml: TB_META_PASSWORD / TB_INGEST_PASSWORD via the same generate-once, upgrade-stable 3-tier resolution as TB_CREDMGR_PASSWORD (operator pin -> existing Secret value -> randAlphaNum), alphanumeric-pin validation, emitted only when serviceDbAccounts is on. - jobs-manager-deployment: SERVICE_DB_ACCOUNTS=1 + TB_META_USER/TB_INGEST_USER + the two secretKeyRef passwords, rendered only when the flag is on. Gated so a default install is byte-identical (verified: `helm template` with the flag off renders none of the new keys). helm-unittest coverage added to secrets_test.yaml and jobs_manager_test.yaml (49 assertions, green). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * sec(#1528): bump chart version 1.9.27 -> 1.9.28 (chart-version-guard) templates/values/schema changed, so the chart-version guard requires a version: bump — a Helm repo only publishes a new version, and an unbumped edit ships dark or overwrites a published tarball. Additive feature => patch bump. appVersion unchanged (wiring only, no app image change). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#1528): bump appVersion to 1.9.28 in lockstep with version (Bugbot) .cursor/BUGBOT.md flags a Chart.yaml version bump without the matching appVersion — app.kubernetes.io/version derives from appVersion, so leaving it at 1.9.27 would tag upgraded resources as the previous release. Every prior bump moved the two in lockstep; this restores that. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…#1492) (#645) * ci: arm the action-pin gate — findings fail, not just report (backend#1492) The pinning work is done: 415 mutable action refs across the fleet on 2026-08-04, zero on every develop branch today (#1490, #1491 closed with that scan). This makes it stay done. action-pins has been running here since it reached main, but in soft-fail: it reports a violation and lets the PR merge anyway. That is the state hand-pinning already proved insufficient. While #1446 was open to pin one action, #1449 added a SECOND unpinned call site of the SAME action in a non-overlapping hunk of the same file. Both merged cleanly, git reported no conflict because there was none, and it was caught by a human reading the diff rather than by any gate. That incident is why #1492 exists. Arming imports no backlog. Measured immediately before this change, across all 15 develop branches carrying this caller: ZERO pin violations. So this flips a check that is already green into one that stays green. What it enforces, per D10: third-party and actions/* refs pinned to a full 40-character commit SHA with a trailing exact-version comment; tracebloc/* refs pinned to @main (Q3). A floating tag can be repointed by its owner at any commit, which is the supply-chain risk the whole class exists to remove. The .github repo armed this first and for a specific reason: it publishes these workflows to every other repo, so its own refs are inherited fleet-wide (.github#178, backend#1603). * fix(ci): put action-pins in the with: block, not workflow_dispatch inputs My first commit anchored on the bare key `all-files:`, which matched its FIRST occurrence in the file - the workflow_dispatch INPUT DEFINITION, not the code-quality call's `with:` map. So `action-pins: true` landed as an invalid input declaration (inputs need description/type), the workflow failed to load, and code-quality did not run at all on this PR. Measured: 10 of 15 callers hit this; the 5 that did not are the ones whose first `all-files:` happens to be in `with:`. Now anchored on the `with:` that follows the code-quality `uses:` line, so the insertion point is structural rather than a name that appears twice. Validated by parsing the result: action-pins present in jobs.quality.with, absent from workflow_dispatch.inputs. Also restores the trailing newline the first commit stripped.
LukasWodka
commented
Aug 10, 2026
ContributorAuthor
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit fcadee6. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated promotion by the release train (RFC-0008 D14). Head is the train-managed
release-train/to-stagingbranch (a mirror ofdevelop), so it never collides with a human PR. Merged only when the fr-gate is green.Note
Medium Risk
Wide staging promotion touching CronJob behavior, pod resources, and optional DB credential provisioning; defaults stay off for new flags but chart upgrades will change jobs-manager requests and refresh logic fleet-wide on upgrade.
Overview
Automated develop→staging promotion (chart 1.9.22 → 1.9.28) that rolls up several operational and security fixes rather than a single feature.
Client Helm chart: The image-refresh CronJob script now builds an explicit in-cluster kubeconfig (fixes fleet-wide
localhost:8080kubectl failures), adds flap detection after consecutive failed rollouts (maxRefreshAttempts), and tightens fail-closed annotation reads plus request timeouts on API calls. Auto-upgrade Jobs getactiveDeadlineSecondsderived from the Helm timeout via a newtracebloc.durationSecondshelper so hunghelm repofetches cannot block future ticks. jobs-manager default memory/cpu requests are raised and requests.memory is pinned to limits to reduce OOM kills during mass restarts; requests-proxy gains TCP probes on :8888. An opt-inserviceDbAccountsflag (default off) provisionstb_meta/tb_ingestsecrets and env for jobs-manager (RFC-0003 S1, no consumers until S2).CI: Code-quality workflow enables hard-fail action pin enforcement.
Installers / images: Default readiness and API wait budgets increase (300s → 600s / 180s tunable); macOS setup uses bounded Homebrew/Colima/Docker installs. k3s-cuda Dockerfile fixes
COPY --excludeordering and build.sh verifies the rootfs before push.Tests add regression locks for kubeconfig preamble, service DB secrets, and updated resource defaults.
Reviewed by Cursor Bugbot for commit fcadee6. Bugbot is set up for automated code reviews on this repo. Configure here.