Add request proxy url to jobs manager yaml file by saqlainsyed007 · Pull Request #119 · tracebloc/client

saqlainsyed007 · 2026-05-12T12:35:28Z

Summary

Type of change

Test plan

Screenshots / recordings

Deployment notes

Checklist

Tests added / updated and passing locally
Docs updated if behavior or config changed
No secrets / credentials in the diff
For security-sensitive paths: appropriate reviewer requested

Note

Low Risk
Low risk Helm template change that only adds a new environment variable to the jobs-manager container; main risk is misconfiguration if the service URL is wrong or unavailable at runtime.

Overview
Adds REQUESTS_PROXY_URL (set to http://requests-proxy-service:8888) to the jobs-manager Deployment Helm template so the manager can be configured to reach the requests-proxy service.

^{Reviewed by Cursor Bugbot for commit 273835a. Bugbot is set up for automated code reviews on this repo. Configure here.}

* Merge pull request #88 from tracebloc/ci/add-wip-limit-caller ci: add WIP-limit-check caller workflow * feat(requests-proxy): register requests-proxy in Helm chart (#95) * feat(requests-proxy): register requests-proxy in Helm chart - Add requests-proxy Deployment and Service templates - Auto-generate requests-proxy-admin token on first install (preserved across upgrades via lookup; override with requestsProxyAdminToken) - Inject REQUESTS_PROXY_ADMIN_TOKEN into jobs-manager via the same secret - Add images.requestsProxy and resources.requestsProxy values Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update order of setting request proxy admin token * Bugbot Fix YAML * Bugbot fix add validation for request proxy --------- Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Merge pull request #106 from tracebloc/docs/drop-stale-helm-charts-refs-105 docs: drop stale tracebloc-helm-charts references in INSTALL.md * ci: add FR-pass comment caller for multi-stage kanban flow * ci: add FR gate caller for staging/main promotions * chore: sync main → develop after misrouted docs PRs (#108) * docs: fix README Deploy section (Helm not docker), surface in-repo docs The Deploy section opened with `docker pull tracebloc/client:latest`, but this repo ships a Helm chart — the actual install is `helm install`. External walkthrough URLs (`/local-linux`, `/local-macos`, `/aws`, `/deployment-overview`) didn't match any path in the tracebloc/docs tree, so they 404. The in-repo documentation (`docs/INSTALL.md`, `docs/MIGRATIONS.md`, `docs/migration-tools/README.md`, `client/MIGRATION.md`) was never linked from the README despite being the operational source of truth. Surgical change — the rest of the README stays as-is: - Replace `docker pull` with `helm repo add` + `helm install` (matches docs/INSTALL.md) - Call out chart version (v1.3.1) and platform support (AKS / EKS / bare-metal / OpenShift) up front - Table linking every in-repo operational doc - Fix external URLs to match actual tracebloc/docs paths (local-deployment-guide-linux, local-deployment-guide-macos, eks-client-deployment-guide, azure-deployment-guide) - Pull NetworkPolicy/CNI prerequisite into a callout Closes#101 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: surface standalone installer in README and INSTALL.md The standalone installer (bash <(curl -fsSL tracebloc.io/i.sh) / irm tracebloc.io/i.ps1 | iex) is the one-command path for evaluation, local dev, and first-time installs — it provisions a cluster, detects GPU drivers, and deploys the client. Today it isn't documented anywhere reachable from this repo, so readers see the multi-step helm install flow as the only option. README: - New "Quick install" subsection at the top of Deploy with macOS/Linux and Windows commands, brief description of what it does, and a pointer to the local helper scripts under scripts/ - Existing helm flow relabeled as "Helm install (production)" — now positioned as the option for existing production clusters docs/INSTALL.md: - Top-of-doc callout pointing at the standalone installer for non-production users - Production-focused content untouched Closes#103 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: reframe Quick install — same client, different cluster path Previous wording ("Best for evaluation, local dev, and first-time installs" / "Just trying it out? For local dev or a quick evaluation") implied the standalone installer produces a lesser/demo client. It doesn't — it produces the same full client, just on a cluster the script provisions for you. Reframes the differentiator around cluster ownership instead of install quality: - README: "Use this when you don't already have a cluster — the result is a full client install, not a demo." Helm subsection retitled from "Helm install (production)" to just "Helm install" with "For existing Kubernetes clusters". - INSTALL.md: callout opens with "Don't have a Kubernetes cluster yet?" and emphasizes "a full tracebloc client". Refs #103 * docs: explicit https:// on installer URLs (security) curl and PowerShell's irm both default to HTTP when no scheme is specified, so `curl -fsSL tracebloc.io/i.sh` and `irm tracebloc.io/i.ps1` issue plaintext requests. The downloaded body is piped straight into bash / iex, so a network-level attacker between the user and tracebloc.io could MITM the response and inject arbitrary code. Add explicit `https://` to every installer URL in README.md and docs/INSTALL.md so the request is encrypted from the first byte. Refs #103 * ci: bootstrap FR-pass caller on main * ci: bootstrap FR gate caller on main --------- Co-authored-by: Lukas Wuttke <lukas@tracebloc.io> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: lukasWuttke <54042461+LukasWodka@users.noreply.github.com> * chore(auto-upgrade): run cronjob hourly at :23 (#112) Switches the auto-upgrade CronJob default schedule from "23 2 * * *" (daily 02:23 UTC) to "23 * * * *" (hourly at :23). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Merge pull request #115 from tracebloc/chore/bump-chart-1.3.2-develop chore(client): bump chart 1.3.1 -> 1.3.2 (develop sync) * ci: drop push-tags trigger from release-helm-chart workflow (#117) * ci: drop push-tags trigger from release-helm-chart workflow `gh release create v<x.y.z>` (the established release path per `gh release list`) fires both `push` (tag) and `release` (published) events, which causes two parallel workflow runs to race for the gh-pages push. The slower run fails with non-fast-forward. Most recent example: v1.3.2 cut today — run 25492826437 (release event) failed; run 25492826350 (push event) succeeded. Artifacts landed fine, but the failed sibling shows up as a red X on the release and is noise for anyone debugging future releases. Keeping only `release: published` removes the race. The `Upload chart to GitHub Release (on tag)` step's `startsWith(github.ref, 'refs/tags/')` guard still evaluates true for release events (`github.ref` is the tag ref), so the upload step behaviour is preserved. Closes#116 * ci: harden release-asset upload against actions/runner#2788 With the push-tags trigger removed, the upload step's `if: startsWith(github.ref, 'refs/tags/')` guard is the only thing keeping the upload from running, but it silently evaluates to false when `github.ref` arrives empty — a known intermittent runner bug (actions/runner#2788, still open as of 2026-05). The same bug also affects `github.ref_name`, which softprops/action-gh-release@v2 uses by default to derive the tag, so the action itself can target the wrong release (or fail) when the bug fires. Drop the now-redundant `if:` guard (the workflow only runs on `release: published`, so every run is by definition a release event) and pass `tag_name` explicitly from the release event payload, which is unaffected by the bug. * ci: pin checkout ref to release tag (actions/runner#2788 hardening) actions/checkout@v4 defaults `ref` to github.ref, which is the same field hit by actions/runner#2788 — the still-open intermittent bug where github.ref arrives empty on release-triggered runs. Per the action's docs, when "checking out the repository that triggered a workflow, this defaults to the reference or SHA for that event. Otherwise, uses the default branch." So an empty github.ref would fall back to the repo default branch (develop here), and we'd package the chart from develop's HEAD instead of the tagged commit. Pin ref explicitly to github.event.release.tag_name, which is part of the release event payload and is unaffected by the runner bug. * Add MySQL Host to request proxy yaml file (#118) Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local> * Add request proxy url to jobs manager yaml file (#119) Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local> * Remove REQUESTS_PROXY_ADMIN_TOKEN (#120) Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local> * Reduce dependency on values.yaml file for requests proxy (#122) Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local> * feat(#86): ingestor Helm subchart + companion RBAC/service/authz for new ingestion endpoint (#123) * feat: companion chart changes for ingestion endpoint (client-runtime#21) Wires the cluster side of the new ingestion flow into the main client chart so the upcoming ingestor subchart can actually reach jobs-manager. Five small changes: 1. **rbac.yaml** — adds three permissions to jobs-manager's RBAC: - authentication.k8s.io/tokenreviews create - configmaps create - secrets create The endpoint validates caller SA tokens via TokenReview and creates a per-run ConfigMap (ingest.yaml) + Secret (BACKEND_TOKEN) before spawning the ingestor Job. `tokenreviews` is cluster-scoped and only added to the ClusterRole branch; customers with `clusterScope: false` won't have the ingestion endpoint authenticate. Documented in the rule comments. 2. **jobs-manager-service.yaml** (new) — ClusterIP exposing port 8080 at the stable name `jobs-manager`, so the ingestor subchart's post-install hook doesn't need to discover Pod IPs. 3. **jobs-manager-deployment.yaml** — adds containerPort 8080 on the `api` container, mounts the ingestion-authz ConfigMap at `/etc/tracebloc/ingestion-authz.yaml`, declares the corresponding pod-level volume. 4. **ingestion-authz-configmap.yaml** (new) — renders the `ingestionAuthz.allowed` policy customers configure in values.yaml. Mounted into jobs-manager and read at startup by `submit_ingestion_run.load_authz_policy`. Each entry maps (namespace, service_account) → allowed table_prefixes; omitted `namespace` defaults to .Release.Namespace. 5. **values.yaml** — adds the `ingestionAuthz.allowed` default that permits the ingestor subchart's default SA (named `ingestor`) to ingest into any table. Customers tighten via overrides. Verified ──────── - helm lint passes (only pre-existing icon-recommended INFO). - helm template renders all five resources cleanly with expected values (Service name, RBAC verbs, container port, volume mount). - helm unittest: 116/116 tests pass (existing snapshots unchanged). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(#86): ingestor Helm subchart (post-install hook submits to jobs-manager) The customer-facing chart that finally closes the end-to-end loop: helm install my-dataset tracebloc/ingestor --namespace tracebloc \ --set-file ingestConfig=./my-ingest.yaml \ --set image.digest=sha256:<digest> Renders the customer's ingest.yaml into a ConfigMap, then a post-install hook Job POSTs `{ingest_config, idempotency_key, image_digest}` to jobs-manager's `/internal/submit-ingestion-run` endpoint (client-runtime#21). jobs-manager validates the SA token via TokenReview, validates the YAML against ingest.v1, mints a backend token, creates the per-run ConfigMap + Secret + Job, returns 201 (or 200 on replay). Layout ────── ingestor/ ├── Chart.yaml appVersion: 0.3.0-rc1 (the data-ingestors release) ├── values.yaml ingestConfig (required, --set-file), image.digest │ (required, sha256), jobsManager.endpoint, │ serviceAccount.create, hook resources, idempotency ├── README.md ownership boundaries + verification commands ├── .helmignore └── templates/ ├── _helpers.tpl ├── serviceaccount.yaml default name "ingestor" ├── configmap-ingest-config.yaml hook-weight 0 └── post-install-job.yaml hook-weight 1, runs as the SA, reads its own token, POSTs. Ownership boundary ────────────────── Per #86's acceptance criteria, the README spells out what `helm uninstall` does and doesn't clean up: This chart owns: ConfigMap (ingest.yaml), the hook Job, the SA. jobs-manager owns: the per-run ConfigMap, Secret, ingestor Job. The cluster owns: the ingested data + metadata POSTed to the backend. `helm uninstall my-dataset` removes only the chart's footprint. The running ingestor Job and its data persist. This is deliberate — uninstall is not a cancel button. The README documents the kubectl command to cancel a run if needed. Implementation choices ────────────────────── - **post-install hook, not a long-lived resource.** The hook is the whole point of this chart — fire once, exit. - **automountServiceAccountToken: true** for the hook Job. That's the whole authentication mechanism — TokenReview on the SA token. Every other tracebloc workload disables automount; this one needs it. - **`hook-delete-policy: before-hook-creation`**, NOT `hook-succeeded`. Keeps the completed Job around so operators can `kubectl logs` the POST response after install. Cleaned up only on the next install under the same release. - **curlimages/curl** as the hook image — small, official, and ships python3 which we use to JSON-encode the multi-line YAML body safely (jq has a JSON-escape edge case for YAML newlines that's easier to side-step than handle). - **idempotencyKey defaults to `<release>-<revision>`** so a `helm upgrade` submits a fresh run. Customers override to a stable UUID if they want strict at-most-once across reinstalls. Verified ──────── - helm lint passes. - helm template renders all four resources (ConfigMap, Job, SA, and the inline templates expand cleanly with --set-file ingestConfig). - Required-value gates fire correctly: missing image.digest fails template; missing ingestConfig fails template. Closes#86 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(#86): pre-render JSON body in ConfigMap, drop python3 + shell JSON escape Three bugbot findings on the first ingestor-chart pass, all real: 1. HIGH — curlimages/curl runtime layer doesn't include python3 (only in the build stage; stripped in the final image). The hook's `python3 -c ...` JSON encoder would fail with "python3: not found" on every install. 2. HIGH — even if python3 were available, the shell syntax `python3 -c "..." VAR=value` puts the assignments AFTER the command, which makes them positional argv, not env. The `os.environ['INGEST_CONFIG']` lookup would raise KeyError. 3. MEDIUM — `nindent 4` after literal template-source indentation puts a leading blank line into the YAML block scalar, so the customer's ingest.yaml gets a "\n" prefix that block-scalar parsers tolerate but is wrong. Structural fix rather than tweaking the script: the three POST-body fields (ingest_config, idempotency_key, image_digest) are ALL known at helm-template time. Render the JSON body in the ConfigMap as `body.json` using Helm's `toJson` filter — which handles multi-line string escaping correctly — then the hook becomes a one-line `curl --data-binary @body.json`. No python3 needed, no shell-side JSON construction at all. Eliminates both HIGH bugs as a category, not just instance-by-instance. For bug 3: use the left-trim action delimiter (dash inside braces) before the `required ... | nindent 4` action so it eats the leading whitespace cleanly. Verified via `helm template` that the rendered `ingest.yaml` now starts cleanly with `apiVersion:`. Verified ──────── - helm lint passes on both client/ and ingestor/. - helm template renders the JSON body with correct escaping (multi-line YAML → "\n"-escaped scalar in JSON). - helm template renders ingest.yaml with no leading blank line. - helm unittest client/: 116/116 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(#86): track ingestor/values.yaml (was silently .gitignored) bugbot caught a serious oversight: `ingestor/values.yaml` exists in the working tree but never made it into the repository. Every `git add ingestor/` silently dropped it because the repo's .gitignore at line 119 has `/*/values*.yaml` — an anti-leak pattern for operator values files — which matches `ingestor/values.yaml`. Without the file the chart is broken on `helm install`: every template references `.Values.hookImage.repository`, `.Values.jobsManager.endpoint`, etc., and Helm renders nil-pointer errors when the keys are absent. Two-line fix: - Add `!ingestor/values.yaml` to .gitignore (mirrors the existing `!client/values*.yaml` exception for the main chart). Documents *why* the exception exists, so a future cleanup pass doesn't re-introduce the bug. - Commit the actual values.yaml file with the defaults already referenced by the README and the templates. Local verification before pushing: helm template my-dataset ingestor/ --namespace tracebloc \ --set ingestConfig=... --set image.digest=sha256:... \ # renders ServiceAccount, ConfigMap, Job correctly. Lesson for future runs: `git add <dir>/` is *not* a verification that files were added — gitignore patterns can silently drop them. Should have verified with `git status` before commit; would have caught this before bugbot did. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: nil-guard ingestionAuthz access for --reuse-values upgrade path (#124) #123's ingestion-authz ConfigMap template did unguarded nested access: {{- range .Values.ingestionAuthz.allowed }} This crashes with "nil pointer evaluating interface {}.allowed" when `.Values.ingestionAuthz` is absent — which is exactly what `helm upgrade --reuse-values` produces against a pre-#123 release. The stored values from the previous deploy don't have the key, and `--reuse-values` doesn't pick up new chart defaults, so the upgrade fails before any of the new resources are created. A real user hit this immediately after #123 merged: Error: UPGRADE FAILED: template: client/templates/ ingestion-authz-configmap.yaml:20:21: executing "..." at <.Values.ingestionAuthz.allowed>: nil pointer evaluating interface {}.allowed Fix: collapse the missing-parent and missing-child cases to an empty list with `default dict` + `default list`. The rendered ConfigMap becomes `allowed:` (empty), which the authz policy parser treats as "no SAs authorized" — fail-safe, matches the intent of "operator hasn't configured this yet". The recommended `helm upgrade` recipe is still `--reset-then-reuse-values` (picks up new defaults including the non-empty `ingestionAuthz.allowed` default), but the template no longer requires that — it renders correctly under either path. Verified ──────── - helm template renders cleanly with default values (full policy), with `--set ingestionAuthz=null` (empty allowed list), and with `--set ingestionAuthz.allowed=null` (same). - helm unittest client/: 116/116 pass, no snapshot changes. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(#125): wire INGESTOR_IMAGE_DIGEST; drop digest requirement from ingestor subchart (#126) * feat(#125): wire INGESTOR_IMAGE_DIGEST; drop digest requirement from ingestor subchart Companion to tracebloc/client-runtime#41 (which made the endpoint treat the request body's `image_digest` as an optional override of a cluster-configured default). With this PR the ingestor image fits the same auto-update model as every other component in the chart: client/values.yaml + images.ingestor.digest: "" The auto-upgrade cronjob bumps this when a new chart version is published; jobs-manager re-rolls and the new env takes effect. client/templates/jobs-manager-deployment.yaml + INGESTOR_IMAGE_DIGEST env, nil-guarded for --reuse-values from a pre-this-PR release. Empty value renders cleanly (no nil pointer), endpoint then accepts only request-body overrides until the operator sets the chart value. ingestor/values.yaml + templates/configmap-ingest-config.yaml + image.digest is now an OPTIONAL override, not required. + body.json renders without `image_digest` when none is set; the key is included only when the customer explicitly pinned via --set image.digest=... (the override path: reproducing old runs, testing pre-rollout versions, air-gapped mirrors). ingestor/README.md + Removes image.digest from "Required values". + Adds "Pinning a specific image version" section explaining the override use cases and when to reach for them. + Top-of-README install snippet drops --set image.digest=... — the dominant path is now `helm install --set-file ingestConfig=...`. Once both PRs land, the bootstrap step is a one-line bump of client/values.yaml's images.ingestor.digest to the current ghcr.io/tracebloc/ingestor release digest, plus a chart version bump so the auto-upgrade cronjob promotes it. Future ingestor releases follow the same pattern — bump digest + chart version, customers' auto-upgrade picks it up on the next tick. Verified ──────── - helm lint passes on both charts. - helm template renders: - env populated when images.ingestor.digest is set - env empty (nil-guard) when images.ingestor key absent entirely (simulates --reuse-values from pre-this-PR release) - body.json without image_digest when no override - body.json with image_digest when explicit --set image.digest=... - helm unittest client/: 116/116 pass. Closes#125 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bootstrap ingestor digest + bump chart version 1.3.2 → 1.3.3 Activates the auto-update model introduced by the rest of this PR. Without the value set, jobs-manager runs with `INGESTOR_IMAGE_DIGEST=""` and the ingestion endpoint returns 503 for every call that doesn't include a body override — which is the *opposite* of the "customer doesn't have to think about digests" UX this PR is supposed to enable. Two coupled bumps: client/Chart.yaml version: 1.3.2 → 1.3.3 appVersion: 1.3.2 → 1.3.3 Required for the auto-upgrade cronjob to detect this release. `helm search repo` orders by version; without a bump customers stay on 1.3.2 and never see the new env wiring. client/values.yaml images.ingestor.digest = "sha256:e6639b084d0d377072dc908db376050914ebd49c730ddaa13f838d10f5482ea9" The data-ingestors v0.3.0-rc1 release. Future ingestor releases bump both this and Chart.yaml's version; eventually a workflow in tracebloc/data-ingestors can raise the PR automatically when a new image is published. After this lands and the chart is published to gh-pages, a `helm upgrade --reset-then-reuse-values` on the customer's cluster (or the daily auto-upgrade cronjob's next tick) rolls jobs-manager with the env populated, and `helm install tracebloc/ingestor --set-file ingestConfig=...` — no `--set image.digest=...` — works. Verified ──────── - helm lint client/ clean. - helm template shows INGESTOR_IMAGE_DIGEST env populated with the real digest. - helm unittest client/: 116/116 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(#127): ingestor chart auto-resolves jobs-manager endpoint to release namespace (#128) The ingestor subchart's default jobsManager.endpoint hardcoded "tracebloc" as the parent release's namespace: http://jobs-manager.tracebloc.svc.cluster.local:8080 Any release in a non-"tracebloc" namespace failed the post-install hook with `curl: (6) Could not resolve host: …`, blocking end-to-end ingestion. Surfaced today during real-cluster validation on a release deployed to `tracebloc-templates`. Fix shape: leave the values.yaml default empty; have the post-install hook template the endpoint to use `.Release.Namespace` when no value is set. The override path (cross-namespace install) keeps working — set `jobsManager.endpoint` explicitly and it wins over the default. values.yaml jobsManager.endpoint: "" (was hardcoded to tracebloc namespace) + comment explaining the auto-resolve + override semantics templates/post-install-job.yaml JOBS_MANAGER_ENDPOINT defaults to http://jobs-manager.<.Release.Namespace>.svc.cluster.local:8080 when .Values.jobsManager.endpoint is empty. README.md Frequently-overridden-values entry corrected. Verified ──────── - helm template into namespace `tracebloc-templates` → http://jobs-manager.tracebloc-templates.svc.cluster.local:8080 - helm template into namespace `some-other-ns` → http://jobs-manager.some-other-ns.svc.cluster.local:8080 - helm template with --set jobsManager.endpoint=http://port-forward.localhost:8888 → wins over the default. - helm lint clean. Closes#127 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(#129): parent client chart owns the shared ingestor ServiceAccount (#131) The ingestor ServiceAccount is shared by every `tracebloc/ingestor` subchart release in a namespace, but it was owned by the first such release. Concurrent installs of a second ingestor release collided with Helm's "cannot import into current release"; uninstalling the first release ripped the SA out from under all the others. Move the SA into this parent chart, which already owns the matching `ingestionAuthz` ConfigMap, so the SA + policy have the same lifecycle and every ingestor release in the namespace shares the SA cleanly. Plumb the name through `ingestionAuthz.serviceAccountName` as a single source of truth — both the new SA template and the default `allowed` entry in the authz ConfigMap dereference it via the new `tracebloc.ingestorServiceAccountName` helper. The helper nil-guards pre-#129 `--reuse-values` upgrades by defaulting to "ingestor". Document the SA adoption path in `client/MIGRATION.md` for clusters that already have an `ingestor` SA owned by a 0.1.0 subchart release — re-annotate before upgrading the parent chart so Helm doesn't refuse the import. Bumps chart to 1.3.4. Pair with tracebloc/ingestor 0.2.0, which flips `serviceAccount.create` default to `false` so subchart releases stop trying to own the SA themselves. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(#130): default idempotency key to install-time stamp, not release revision (#132) `ingestor.idempotencyKey` previously fell back to `<release>-<revision>` when `.Values.idempotencyKey` was unset. Helm restarts revisions at 1 after `helm uninstall`, so reinstalling under the same release name produced the same key. If anything dedupe-relevant changed in between (image digest is the dominant case during testing), jobs-manager correctly rejected the second submission with a 409 — but to a customer following the README it looked like the chart was broken. Default to `<release>-<unix-epoch>` instead. Each install gets a fresh key; the opt-in stable-UUID path remains for callers who actually want at-most-once semantics across reinstalls. Note on the printf format: Sprig's `unixEpoch` returns a string (not an int), so the formatter is `%s-%s`, not `%s-%d`. Bumps ingestor subchart 0.1.0 → 0.1.1 (default-behavior change). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(#129)!: default serviceAccount.create=false; parent chart owns the SA (#133) The ingestor SA is shared across every `tracebloc/ingestor` release in the namespace. The previous per-release ownership made the second concurrent install collide with Helm's "cannot import into current release" error, and uninstalling the first release deleted the SA out from under any sibling release that worked around the collision with `serviceAccount.create=false`. The parent `tracebloc/client` chart 1.3.4 now owns the SA, exposing its name via `ingestionAuthz.serviceAccountName`. This subchart's default flips to `create: false` so it consumes that shared SA. The `name` value is still required so the post-install hook Job knows which SA's token to mount. `serviceAccount.create=true` remains available as an escape hatch for operators on a pre-1.3.4 parent chart, with a comment in values.yaml explaining when (and only when) to flip it back on. Breaking change: bumps chart to 0.2.0. Pair with the 1.3.4 parent chart bump; see the parent's MIGRATION.md "Upgrading to 1.3.4" section for the SA-adoption procedure on clusters where a 0.1.0 release already created the SA. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(chart): bump ingestor digest to v0.3.0 + chart to 1.3.5 (#134) v0.3.0 is the first production-ready ingestor release (signed + SBOM), validated end-to-end against EKS on 2026-05-19 (6 files in PVC + 576 MySQL rows via the declarative chart path). The previous default (v0.3.0-rc1) had three real-cluster bugs that landed as tracebloc/data-ingestors#106: - #103 wheel + sdist were missing schema/ingest.v1.json - #104 image-resolution validator tuple-vs-list comparison - #105 _has_extension dot/case normalization (no more cat1.jpeg.jpeg) Chart bumped to 1.3.5 so the auto-upgrade cronjob (#69) detects the change and rolls customers onto v0.3.0 on the next tick. ingestor image: ghcr.io/tracebloc/ingestor@sha256:463e2367...07a4a cosign verify available; release notes contain the verification command. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(#135): publish ingestor subchart alongside parent chart (#136) The customer-facing install path is helm repo add tracebloc https://tracebloc.github.io/client helm install my-dataset tracebloc/ingestor \ --namespace tracebloc-templates \ --set-file ingestConfig=./my.yaml For `tracebloc/ingestor` to resolve from that helm repo, the ingestor subchart must be packaged into gh-pages alongside the parent client chart. Before this PR, `release-helm-chart.yaml` only ran `helm package ./client`, so the second install path returned `Error: chart "ingestor" not found`. helm-ci.yaml also only lints the parent chart, so any future regression in `ingestor/templates/` would land on develop without CI noticing. Three changes: 1. release-helm-chart.yaml: package + index BOTH client and ingestor into a single shared index.yaml. Attach both tgzs to the GitHub release for download-by-tag pinning. 2. helm-ci.yaml: lint the ingestor subchart on every PR alongside the per-platform client lints. Plain `helm lint --strict ./ingestor` is enough — its only required value (ingestConfig) emits INFO not FAIL, and the chart's templates don't branch on platform so the per-platform values-file matrix doesn't apply. 3. ingestor/Chart.yaml: bump appVersion 0.3.0-rc1 → 0.3.0 to match the tracebloc/data-ingestors v0.3.0 release that just shipped. Chart version (0.2.0) is unchanged; appVersion is descriptive. Validated locally: both charts package cleanly (client-1.3.5.tgz, ingestor-0.2.0.tgz), all four platform-specific client lints pass, ingestor lint passes. Closes#135. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(ingestor): explain image vs chart update lifecycle (#138) Customers ask: "the cluster has an auto-upgrade cronjob — does that mean my ingestor chart updates too?" The answer is nuanced: the image auto-updates (via INGESTOR_IMAGE_DIGEST on jobs-manager, kept current by the cronjob), but the chart on your workstation is independent — Helm's repo cache doesn't refresh itself. Add a "How updates work" section that explains the two-layer model and the strong property that the image you run is decoupled from the chart version that submitted the request. Plus an explicit FAQ on previously-installed ingestor releases (nothing to upgrade — fire-and-forget). No code change. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix three bugbot findings from PR #137 review (#142) * fix(#139): preserve idempotency key across helm upgrade The ingestor.idempotencyKey helper defaulted to "<release>-<unix-epoch>" and re-stamped on every render. `helm upgrade --reuse-values` preserves the stored value "" (not the previously-rendered key), so the template re-evaluated `now | unixEpoch` and produced a NEW key each upgrade — accidentally creating duplicate ingestion runs from what customers expected to be no-op upgrades. Contradicts the documented behavior in ingestor/README.md added in #138. Look up the existing post-install hook ConfigMap from the previous render and reuse its idempotency_key. On fresh install (or after uninstall) the lookup returns empty and we fall through to the now-based default. `helm template` (no cluster connection) returns empty for lookup too, so local previews still get a fresh key per render — matches the in-cluster install path the first time. Caught by bugbot on PR #137 review. Closes#139. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(#140): read requests-proxy resources from values The requests-proxy deployment hardcoded its container resources, ignoring the resources.requestsProxy schema entry that values.schema.json has defined since the requests-proxy was added. Every other component (jobsManager, podsMonitor, mysql) reads from .Values.resources.<name>.* with defaults — bring requestsProxy in line with that pattern. Adds the resources.requestsProxy block to values.yaml with the existing hardcoded defaults so behavior on a fresh install is unchanged. The template uses the default-through-dict nil-guard idiom so `helm upgrade --reuse-values` from a pre-1.3.6 release (where the value didn't exist) still renders cleanly without crashing on a nil parent. Caught by bugbot on PR #137 review. Closes#140. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(#141): add images.ingestor entry to values.schema.json values.yaml has had images.ingestor.digest since #126, and the jobs-manager template surfaces it as INGESTOR_IMAGE_DIGEST, but the schema didn't validate it — every other image (jobsManager, podsMonitor, resourceMonitor, requestsProxy, mysqlClient, busybox) has an entry. An operator setting --set images.ingestor.digest=foo (not the canonical sha256:<64-hex>) bypassed schema validation and failed only later inside submit_ingestion_run.py. Add the missing entry mirroring the other image entries' shape. helm template now rejects malformed digests at chart-template time ("values don't meet the specifications of the schema(s)... Does not match pattern '^(sha256:[a-f0-9]{64})?$'") rather than waiting for runtime. Caught by bugbot on PR #137 review. Closes#141. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: lukasWuttke <54042461+LukasWodka@users.noreply.github.com> Co-authored-by: Syed Is Saqlain <saqlain.syed007@gmail.com> Co-authored-by: Syed Saqlain <syedsaqlain@MacBook-Pro.local> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Lukas Wuttke <lukas@tracebloc.io>

… diagnostics (#269) §4 ("APPLICATION-pod egress through a proxy", client-runtime#119) was a flaky required check ("E2E auth-proxy (squid)") that intermittently red-X'd develop (~1 in 4; e.g. run 27765964135) and randomly blocked unrelated PRs. Two causes: 1. Silent failure. Under `set -euo pipefail` the diagnostic `grep | sed` lines ran before the real assertion; an empty section made grep exit 1 → pipefail → set -e killed the script with NO output (CI showed only "pod/egress-app created" then "exit code 1"). Append `|| true` so the diagnostics are non-fatal and the assertion fires with its reason. Same footgun fixed in §3. 2. External-network dependency (the real flake). §4 curled the real https://api.tracebloc.io/ through the in-cluster squid, depending on the runner's internet to a production host at test time. Make it hermetic: target a reserved-TLD stand-in host (backend.tracebloc-e2e.test) aliased via hostAliases on both the squid and app pods to the cluster's own kube-apiserver ClusterIP — a guaranteed in-cluster HTTPS:443 listener. The CONNECT tunnel now terminates in-cluster with zero external I/O, preserving the #119 intent (WITH proxy env → CONNECT tunnel via squid; env unset → direct dial). Validated: 3/3 deterministic local passes; both calls hit 10.43.0.1 in-cluster (no api.tracebloc.io reachout). bash -n + shellcheck --severity=error clean. Closes#268 Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

… backend-reachability test, installer hardening) (#273) * Merge pull request #260 from tracebloc/fix/job-resource-defaults-745 test(jobs-manager): lock spawned-job RESOURCE_REQUESTS/LIMITS default at 8Gi * test(e2e-proxy): exercise application-pod egress through the proxy (Charité setup) (#264) * test(e2e-proxy): exercise application-pod egress through the proxy The squid harness proved NODE egress (image pulls) but stopped before any application pod — so it never caught client-runtime#119, where the spawned ingestion Job carried no proxy env and dialled the backend directly. Add a section that runs a pod WITH the ingestion-style proxy env (must traverse the squid to reach the backend) and a pod WITHOUT it (must bypass it / go direct), asserting both against the squid access log. Models the Charité proxy-only setup at the application layer; pairs with the behavioural unit tests on client-runtime#119. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(e2e-proxy): rework app-pod egress to an in-cluster squid Service Running the first version on a real k3d cluster surfaced that a POD cannot resolve host.k3d.internal (it is a node-level alias for image pulls, not pod DNS), so the proxied probe failed with `curl (5) Could not resolve proxy`. Rework: stand up an in-cluster squid Deployment+Service the test pods reach by Service DNS (also a closer model of a real corporate proxy reachable by name), with a readiness probe gating rollout on squid actually listening (fixes the probe-before-bind race seen in the first attempt). A pod WITH the ingestion proxy env must reach the backend through the squid; a pod WITHOUT it must bypass it. Auth survival stays covered by the host-squid sections (1-3). bash -n + shellcheck + embedded-YAML parse all clean; Service-DNS resolution verified locally. Full proxied-curl run is exercised by the e2e-proxy CI job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(e2e-proxy): make app-pod egress assertion deterministic (single pod, curl -v) §4 now uses ONE pod carrying the ingestion-style proxy env that makes two calls to the same backend: WITH the env it must tunnel via the in-cluster squid (a CONNECT tunnel); with the env unset it must dial direct. Proof is taken client-side from `curl -v` (the CONNECT-tunnel lines), not by reading squid's access.log — that file is buffered by the log daemon and came back empty when read right after the probe, producing false failures. Also set BOTH proxy-env cases: curl honours the lower-case `https_proxy` for HTTPS and the upper-case alone is not reliably picked up, so the probe must emit both — exactly as the real ingestion env does. A single pod with a single log also removes the multi-pod scheduling / log-flush races that made the earlier two-pod form flaky. Validated end-to-end on k3d: A (proxy env) -> "Establish HTTP proxy tunnel to api.tracebloc.io:443" + "CONNECT tunnel established, response 200" + 200 OK B (env unset) -> direct connect to the backend IP, no proxy tunnel, 200 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * ci(security): add public-repo PII gate caller (#263) Blocks PRs that leak customer/partner names or secrets in title/body/commits. Calls the reusable gate in tracebloc/.github. Inactive until the org PII_DENYLIST secret is set (warns, doesn't block, until then). * ci: add concurrency cancellation + job timeouts to chart CI (#265) helm-ci.yaml and installer-tests.yaml are the repo's most expensive workflows (a real k3d cluster, a 9-distro docker-in-docker matrix, Windows Pester) but had no concurrency control, so a PR re-push left stale runs burning to completion. Add a per-ref concurrency group that cancels superseded PR runs only (push/schedule runs are never cancelled), matching the pattern already used in client-runtime's tests.yml. Add timeout-minutes to every job so a hung k3d/squid/distro step can't run to the 6h default. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): fail fast when HOST_DATA_DIR is on a network filesystem (#261) Detect NFS/CIFS/SMB for HOST_DATA_DIR in preflight (bash + PowerShell) and fail fast with an actionable message instead of a cryptic MySQL CrashLoopBackOff ~20 min into install: MySQL/InnoDB corrupts on network storage and the chart root chown init-container is blocked by NFS root_squash. - preflight.sh: _pf_fstype reader (findmnt, then GNU stat, then df+mount; portable incl. macOS) + _pf_storage_type wired into run_preflight. Allowlists network fstypes so local FSes including overlay/tmpfs (CI) pass. - install-k8s.ps1: Get-PfFsType (UNC / network drive) + Test-Preflight check. - TRACEBLOC_ALLOW_NETWORK_FS=1 overrides (mirrors TRACEBLOC_ALLOW_ARM64). - Tests: 10 bats cases + Pester cases (network -> fail, override, undetermined, Windows-only Get-PfFsType reader). Part 1 of 3 for tracebloc/backend#743. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(installer,chart): place datasets on a network mount while MySQL stays local (#262) * feat(installer): fail fast when HOST_DATA_DIR is on a network filesystem Detect NFS/CIFS/SMB for HOST_DATA_DIR in preflight (bash + PowerShell) and fail fast with an actionable message instead of a cryptic MySQL CrashLoopBackOff ~20 min into install: MySQL/InnoDB corrupts on network storage and the chart root chown init-container is blocked by NFS root_squash. - preflight.sh: _pf_fstype reader (findmnt, then GNU stat, then df+mount; portable incl. macOS) + _pf_storage_type wired into run_preflight. Allowlists network fstypes so local FSes including overlay/tmpfs (CI) pass. - install-k8s.ps1: Get-PfFsType (UNC / network drive) + Test-Preflight check. - TRACEBLOC_ALLOW_NETWORK_FS=1 overrides (mirrors TRACEBLOC_ALLOW_ARM64). - Tests: 10 bats cases + Pester cases (network -> fail, override, undetermined, Windows-only Get-PfFsType reader). Part 1 of 3 for tracebloc/backend#743. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(installer,chart): place datasets on a network mount while MySQL stays local Storage split for VMs whose real storage is an NFS/CIFS mount (backend#743): the database must stay on local disk (InnoDB over NFS is unsafe) but the large dataset volume can live on the network mount. Chart: - Parameterize the dataset PV hostPath base via hostPath.datasetPath (helper tracebloc.clientDataHostPath). Default /tracebloc keeps it byte-identical; mysql + logs PV paths are unchanged. values.yaml + schema + nil-guard for --reuse-values upgrades. Installer (bash + PowerShell): - New HOST_DATASET_DIR: validated (must exist + be writable; MAY live outside $HOME unlike HOST_DATA_DIR; system paths barred), bind-mounted into k3d at a distinct /tracebloc-data path; the dataset dir is created there while mysql + logs stay local. When set, the generated values set hostPath.datasetPath=/tracebloc-data and (Linux) pass HOST_UID/HOST_GID env to jobs-manager so spawned ingestion pods write the host-owned NFS export as the owning uid. Preflight notes the dataset dir is exempt from the network-FS block. Tests: new shared_images_pvc_test.yaml + mysql/logs split-only guards (helm-unittest, 259 pass); HOST_DATASET_DIR validation, second-mount, dir-split and values-generation cases (bats). Docs: INSTALL.md checklist + SECURITY.md 5.4. Part 2/3 of backend#743. The end-to-end NFS write path also needs the client-runtime ingestor-uid change (separate PR) so jobs-manager reads HOST_UID. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(#262): fail fast when an existing cluster lacks the dataset bind mount (Bugbot) The HOST_DATASET_DIR -> /tracebloc-data bind mount is baked into the k3d nodes at create time (_create_new_cluster / the PS1 equivalent). k3d cannot add a mount to a RUNNING cluster, but install-client-helm.sh still wrote `datasetPath: /tracebloc-data` into the generated values whenever HOST_DATASET_DIR was merely set — so an existing-cluster re-run pointed the chart's dataset PV at ephemeral in-node storage, silently putting datasets on disposable storage instead of the network export (lost on a restart). Add _check_existing_cluster_dataset_mount (cluster.sh) + the PowerShell equivalent, mirroring the existing _check_existing_cluster_proxy/bind drift checks: on an existing cluster with HOST_DATASET_DIR set, inspect the server node for the /tracebloc-data mount and FAIL FAST with the recreate remedy if it is absent — rather than installing a quietly misrouted dataset volume. Fail-fast (not warn) because this is silent data loss, consistent with the network-FS fail-fast guard. Values generation needs no change: the install now stops in Step 2, before helm runs. +4 bats (cluster.bats): unset -> no-op, mount present -> pass, mount ABSENT -> fail fast, inspect fails -> no-op. bash + shellcheck clean; pwsh parses the .ps1; full cluster suite 27/27. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(installer): non-interactive credentials + softer connect copy (#834) (#266) RFC-0001 Phase 0 (no backend dependency): - Accept TRACEBLOC_CLIENT_ID / TRACEBLOC_CLIENT_PASSWORD so CI / automation / golden images can provision without typing the secret inline. Verified the same way as the prompt (verify_credentials); a bad credential fails the install (no re-prompt in non-interactive mode). The interactive prompt path is unchanged — just wrapped in the else branch. - Soften the "to connect this machine you NEED a tracebloc client / create one" copy to "already have one? enter it (or set the env vars) / need one? create it" — so we stop framing client-creation as a mandatory pre-step (browser sign-in lands in Phase 1). Tests: two new bats cases (env path -> non-interactive write + helm with no prompt; rejected env creds -> error, no helm). The interactive flow tests (re-prompt / inactive / unverified / defaults / max-attempts / one-client guard) still pass unchanged. NOTE: bats #16 (_extract_yaml_value single-quote '' un-escape) fails locally on macOS bash — pre-existing and untouched by this PR (the diff doesn't go near that function); flagging for confirmation against CI. Part of backend#830. Closes#834. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Asad Iqbal <asad.dsoft@gmail.com> * test(e2e-proxy): deflake §4 app-egress — hermetic target + non-silent diagnostics (#269) §4 ("APPLICATION-pod egress through a proxy", client-runtime#119) was a flaky required check ("E2E auth-proxy (squid)") that intermittently red-X'd develop (~1 in 4; e.g. run 27765964135) and randomly blocked unrelated PRs. Two causes: 1. Silent failure. Under `set -euo pipefail` the diagnostic `grep | sed` lines ran before the real assertion; an empty section made grep exit 1 → pipefail → set -e killed the script with NO output (CI showed only "pod/egress-app created" then "exit code 1"). Append `|| true` so the diagnostics are non-fatal and the assertion fires with its reason. Same footgun fixed in §3. 2. External-network dependency (the real flake). §4 curled the real https://api.tracebloc.io/ through the in-cluster squid, depending on the runner's internet to a production host at test time. Make it hermetic: target a reserved-TLD stand-in host (backend.tracebloc-e2e.test) aliased via hostAliases on both the squid and app pods to the cluster's own kube-apiserver ClusterIP — a guaranteed in-cluster HTTPS:443 listener. The CONNECT tunnel now terminates in-cluster with zero external I/O, preserving the #119 intent (WITH proxy env → CONNECT tunnel via squid; env unset → direct dial). Validated: 3/3 deterministic local passes; both calls hit 10.43.0.1 in-cluster (no api.tracebloc.io reachout). bash -n + shellcheck --severity=error clean. Closes#268 Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat(cli#90): in-cluster backend-reachability helm test (WS3) (#270) Adds `egress-reachability-check`, a `helm test` Job that verifies a normal (non-training) pod in the namespace can reach the tracebloc backend API — the egress dependency that gates everything (the cluster authenticates to the backend to obtain its Service Bus credentials, so no backend egress => silent Pending). The required-egress complement to egress-enforcement-check (which verifies the opposite: that training pods are locked out). The probe is deliberately NOT training-labelled (so the lockdown netpol never selects it — it keeps the jobs-manager/requests-proxy egress class) and honours tracebloc.proxyEnv, so it tests the real path. The verdict keys on curl's exit code (TCP reachability), not HTTP status. Run via `helm test <release>`; gated by egressReachabilityCheck.enabled (default true; disable on truly air-gapped clusters). As a test hook it never runs during install/upgrade. Service Bus is intentionally not probed here: its host is fetched post-auth from the backend (static nowhere in the chart) and its egress is brokered by the requests-proxy, whose readiness `tracebloc cluster doctor` already checks. helm-unittest: 6 tests (render / disable / test-hook annotation / not-training- labelled / CLIENT_ENV-driven host / proxy-inherited). Full chart suite green. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(chart): bump client 1.7.1 → 1.8.0 (version + appVersion) (#272) Release v1.8.0 — promotes the 9 commits merged to develop since v1.7.1. Minor bump: #262 (datasets on a network mount while MySQL stays local) is a real PVC-placement change at install time, not inert. Keeps version/appVersion in lockstep so the app.kubernetes.io/version label matches the release. Refs #271 Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(egress-reachability-check): fail closed on TLS/transport errors, not just connect failures (#274) The backend-reachability helm test (#270) keyed success on a denylist of curl exit codes (5/6/7/28) and passed on everything else. But with no --fail, curl exits 0 for any HTTP response, so the only non-zero codes are transport/TLS failures — yet a TLS handshake/cert error (e.g. 35/51/60, typically a proxy intercepting TLS with a CA the cluster doesn't trust) printed "backend reachable" and exited 0: a false pass that defeats the check's purpose. Pass ONLY on curl exit 0 (a full TCP+TLS+HTTP round trip proves the backend is reachable and usable); fail closed on everything else, with a dedicated TLS/cert bucket and a generic catch-all. Companion helm-unittest asserts nothing on the script body; full suite 265/265. Follow-up to #270; flagged by Cursor Bugbot on the v1.8.0 sync PR (#273). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: lukasWuttke <54042461+LukasWodka@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…+ raw log on failure (#346) §4 (the #119 ingestion-egress guard) flaked red on develop (run 29255451968); a re-run of the SAME commit (run 29340987746) passed — a transient, not a #341 regression (#341 never touched this test). Cause: the app pod's section-A curl reaches the in-cluster squid via its Service, but `kubectl rollout status` gates only on the squid pod being Ready, not on its Service endpoints being programmed in kube-proxy. In that window a brand-new pod's first CONNECT to the ClusterIP is refused, and curl gave up before attempting the tunnel (no "Establish HTTP proxy tunnel" line) — so the assertion false-failed. - Add `--retry 5 --retry-connrefused --retry-delay 2` (-m 30) to section A's curl: it rides out the transient connection-refused. A genuine #119 regression still fails all retries, so the guard keeps its teeth. - On the (A) assertion failure, dump the RAW egress-app log + squid pod/endpoints state. This flake surfaced only the filtered B line and a bare "did NOT tunnel"; the raw dump makes any future failure debuggable instead of opaque. bash -n + shellcheck clean. (The k3d e2e can't run locally; CI is the authority.) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…race The §4 app-pod curl used --retry-connrefused, which retries a refused CONNECT (Service endpoints not yet programmed) but NOT a DNS failure. Under CI load the squid Service name isn't yet in the new pod's resolver when curl runs, so it fails with "Could not resolve proxy" (curl exit 5) — which curl does not retry on its own — and §4 flaked red (PR #349 run 29345383969) even though the squid pod and endpoints were up. Add --retry-all-errors (covers the DNS-resolution error too) and bump to --retry 8. A genuine #119 regression still fails all retries, so the guard keeps its teeth. Test-only; not part of the installer supply-chain manifest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… probe (#349) * fix(installer): check cluster is running before the Helm probe in the assess gate Bugbot (High) follow-up: _assess_classify called detect_installed_client (unbounded `helm list -A` / `helm get values`) BEFORE the cluster-servers-running check. On a stopped cluster (post-reboot or manual stop) the k8s API is down, so Helm hangs — violating the gate's "bounded, never-hang" contract — and, when it finally fails, the machine is mislabeled fresh/cluster-no-release, making the cluster-stopped path effectively unreachable on real re-runs. Reorder so the cheap read-only k3d servers-running probe runs first: a stopped cluster short-circuits to degraded/cluster-stopped without ever touching Helm, and detect_installed_client only ever runs against a live API. Adds an ordering guard test asserting the Helm probe is not invoked on a stopped cluster. Manifest regenerated. assess.bats 33/33. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(e2e-proxy): deflake §4 — retry the app-pod proxy-DNS resolution race The §4 app-pod curl used --retry-connrefused, which retries a refused CONNECT (Service endpoints not yet programmed) but NOT a DNS failure. Under CI load the squid Service name isn't yet in the new pod's resolver when curl runs, so it fails with "Could not resolve proxy" (curl exit 5) — which curl does not retry on its own — and §4 flaked red (PR #349 run 29345383969) even though the squid pod and endpoints were up. Add --retry-all-errors (covers the DNS-resolution error too) and bump to --retry 8. A genuine #119 regression still fails all retries, so the guard keeps its teeth. Test-only; not part of the installer supply-chain manifest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…digest pinning (#345) * test(provision): pin the #303 pre-flight's grep contract with hidden `client list` (Refs #141) (#335) The installer's #303 one-client-per-machine pre-flight (`scripts/lib/provision.sh` `_account_owns_namespace`) shells out to `tracebloc client list --plain` and greps the output for `namespace=<ns>([[:space:]]|$)` to refuse a cross-account re-provision. In the cli repo `client list` is now a HIDDEN cobra command — still callable, but nothing here pinned that the pre-flight keeps classifying the CLI's exact --plain output correctly, nor that it still passes --plain. If either drifts the grep silently fails and #303 stops firing. Add two focused unit tests of `_account_owns_namespace` (the consumer half of the cli#141 contract; the producer half is pinned in the cli repo at internal/cli/client_list_contract_test.go). The fixture mirrors, field-for-field, what cli's runClientList prints under --plain: - owned namespace → rc 0; absent → rc 1; a STRICT PREFIX of a real namespace → rc 1 (pins the ([[:space:]]|$) anchor: an account must not "own" acme-prod-0 just because it owns acme-prod-01); - the pre-flight actually invokes the hidden list WITH --plain; - an unreadable list is rc 2 (fall through to create), distinct from rc 1 (refuse) — the distinction #303 branches on. Assertions use `[ -eq ]` + `grep` (real exit codes, robust on bash 3.2); Linux CI (standard-checks.yml → `bats scripts/tests/*.bats` on ubuntu) is the authority. Mutation-verified: dropping the grep anchor or --plain from provision.sh fails these tests. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(proxy): NO_PROXY covers the cloud metadata IP 169.254.169.254 (#337) The auto-augmented NO_PROXY lists (chart tracebloc.proxyEnv helper, bash _augment_no_proxy defaults, Windows Get-EffectiveNoProxy defaults) omitted 169.254.169.254 — behind a corporate proxy, cloud metadata lookups would be routed through the external proxy (broken IMDS access and an unnecessary place for instance credentials to transit). Add the IP to all three lists in lockstep, pin it in the bats + Pester + helm-unittest expectations, regenerate manifest.sha256 for the two touched installer scripts, bump chart 1.9.3 -> 1.9.4. Ref tracebloc/backend#803 (item J), tracebloc/client-runtime#120 Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * feat(installer): stop-and-check gate — recognize an existing install (#339) * feat(installer): stop-and-check gate — hand a healthy re-run to the home screen Re-running the installer on an already-set-up machine no longer drags the user through full provisioning. A new read-only gate (scripts/lib/assess.sh) runs after the banner and classifies the machine: • healthy — cluster running AND a tracebloc release present AND jobs-manager Ready AND the CLI present → print "Already set up on this machine", hand off to `tracebloc` (the home screen), exit 0. • degraded — cluster stopped / workload not Ready / CLI missing / any partial state → print an honest one-liner, fall through to the normal flow to reconcile. • fresh — no cluster, or a cluster with no release → the normal flow. assess is STRICTLY non-mutating and bounded: read-only `k3d cluster list`, `helm list`/`get values` (reuses detect_installed_client), and a bounded `kubectl get` (--request-timeout). On ANY uncertainty it degrades toward the normal flow — never a false "healthy" that would skip a needed install. The hand-off uses `exit 0` (not exec) so the EXIT-trap cleanup still runs; if the CLI is somehow unresolvable it falls back to a status line and still exits 0. --force / --reinstall (or TRACEBLOC_FORCE_REINSTALL=1) bypasses the gate. A healthy machine still short-circuits under curl|bash (output only, no input). Per-layer surgical repair of a degraded machine is a deliberate fast-follow so this PR stays off provision.sh (clear of the active #838 work). Wires the gate into main() (after print_banner, before the roadmap), adds scripts/lib/assess.sh to the bootstrap FILES + gen-manifest, regenerates scripts/manifest.sha256, and adds scripts/tests/assess.bats. New copy says "secure environment", never "client". PR 2 of 2 (PR 1 = the cli home screen, cli#244). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): assess "healthy" = ALL client workloads Ready + guard awk branch Code-review should-fix items on the stop-and-check gate: 1. _assess_cluster_servers_running: add `|| line=""` to the awk-branch assignment, mirroring the jq branch. awk's `exit` closes the pipe, so under `set -o pipefail` a SIGPIPE from k3d (141) — or any k3d failure — would otherwise propagate non-zero out of the assignment and abort the installer. 2. "healthy" must match the installer's OWN definition of ready. The probe now requires ALL the workloads wait_for_client_ready checks — mysql-client, ${ns}-jobs-manager, ${ns}-requests-proxy — not jobs-manager alone. Previously a machine with jobs-manager up but requests-proxy (training egress) or mysql-client down was classified healthy and short-circuited without reconciling — the false-positive we designed against. Single source of truth: the deployment set is extracted into _client_workload_deployments (common.sh); both wait_for_client_ready (summary.sh) and the assess gate consume it, so they can't drift. Renamed _assess_jobs_manager_ready -> _assess_workload_ready (reason stays `workload-not-ready`); any one workload not-Ready/absent -> degraded. Tests: _assess_workload_ready now covers all-three-Ready plus each workload individually down/absent; classify covers "one down -> degraded" via the real probe and "all three Ready + CLI -> healthy". Mutation-checked: shrinking the shared list to jobs-manager-only fails the mysql-client/requests-proxy/one-down tests. Manifest regenerated (assess.sh, common.sh, summary.sh). provision.sh untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): run the helm client install behind a spinner (no more silent ~10-15s) (#333) * fix(installer): run the helm client install behind a spinner (no more silent ~10-15s) Step 4/5 ran `helm upgrade --install` (and the in-place reconcile) blocking, with all output redirected to the log, so the terminal sat frozen ~10-15s (render + apply + image pull) with no feedback. Wrap both in the existing spin_cmd helper: an animated spinner + message, output still streamed to $LOG_FILE, and on failure the log tail to stderr + the same error-exit. Honours RFC-0002 §2 "progress on every wait" (the principle already applied to the CLI's post-submit waits). - bash -n + shellcheck clean; install-client-helm.bats green (33 tests). - Failure path preserved (error-exit + full output in $LOG_FILE). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(scripts): refresh integrity manifest for spinner change gen-manifest.sh after the install-client-helm.sh spinner edit; static-analysis CI gate requires the sha256 manifest to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * chore(chart): prod-only ingestor digest pin (#1028 item 1) (#334) * chore(chart): prod-only ingestor digest pin via values-prod overlay (#1028) Prod deploys the ingestor image by immutable digest; dev/staging keep floating :0.7 (imagePullPolicy=Always). Base values.yaml keeps digest empty; the new client/values-prod.yaml sets only images.ingestor.digest and is layered at prod-release time (-f values.yaml -f values-prod.yaml). Digest verified live against ghcr.io (multi-arch index, amd64+arm64). Adds resolve-ingestor-digest.sh to re-resolve/verify the digest at each cut instead of hand-typing it. Refs tracebloc/backend#1028. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(digest): --write refuses single-arch pin + verifies sed landed Multi-arch guard now inspects the resolved repo@digest and hard-fails under --write instead of only warning (a single-arch pin breaks arm64 and fails helm-ci ingestor-multiarch). --write also greps the overlay for the digest after sed and errors on a silent no-op instead of falsely reporting success. Bugbot findings on client#334. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(digest): read chart tag without yq; fail loudly, never hardcode 0.7 The no-arg TAG default silently fell back to a hardcoded 0.7 when yq was absent, so a prod cut after the chart tag moved could --write a stale/wrong digest while appearing to follow the chart. Reads images.ingestor.tag via yq when present, else a portable awk parse scoped to the images:->ingestor: block; if neither works it errors and exits non-zero. Bugbot follow-up on client#334. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(digest): tolerate pipefail in the multi-arch platforms capture Under set -o pipefail, a failed inspect or a grep -v that filters every line exits non-zero and aborted the whole script even after the digest resolved. Add || true; an empty platforms then trips multiarch=0 so the guard still fires (ERROR under --write, WARNING otherwise). Bugbot round-3 on client#334. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * docs(security): update §8.3 — backend tokens now bounded + revocable (#302) (#342) §8.3 claimed "Backend tokens never expire"; that finding is now mitigated for interactive DS web sessions via a bounded, revocable 30-day ClientAccessToken (backend#933 + frontend-app#575, shipped through backend#590). Document the mitigation, the intentional carve-out for edge devices / bots (which keep the legacy long-lived DRF Token as non-interactive service credentials, by design), and the residual JS-readable-storage risk tracked as the SEC-06 follow-up in tracebloc/backend. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): Windows one-client guard reads clientId via ConvertFrom-Json (#200) (#340) The one-client-per-machine guard regex-scraped `helm get values` YAML and only matched a DOUBLE-QUOTED clientId. Helm re-serializes values on `get`, so typical clientIds come back unquoted -- the guard silently matched nothing and a re-install could re-point the machine to a different Client ID, defeating the protection from #192 (Bugbot finding on #199). Read the values as JSON instead (`helm get values <rel> -n <ns> -o json | ConvertFrom-Json` -> `.clientId`), which sidesteps YAML quoting entirely and matches the adjacent comment that already claimed ConvertFrom-Json. A release with no user values (literal `null`), a missing clientId key, or unparsable output from one release is skipped without aborting the scan of the remaining releases. Tests: existing guard mocks now serve real-helm-shaped output (JSON for `-o json`, unquoted YAML otherwise), plus new cases for unquoted / single- quoted / double-quoted YAML views, a null-values release mid-scan, and values without a clientId key. Pester (Linux container, lts-7.4): 95 passed, 0 failed, 6 skipped ($IsWindows skips). scripts/manifest.sha256 regenerated (install-k8s.ps1 line only) -- the file is on the signed-manifest surface; gen-manifest.sh --check passes. Closes#200 Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * docs(supply-chain): document the Windows bootstrap (install.ps1) (#336) * docs(supply-chain): document the Windows bootstrap (install.ps1) SUPPLY_CHAIN.md scoped itself to install.sh with zero Windows mentions, while the R8 Windows leg (PR #299, released v1.8.5) has shipped the same guarantee for install.ps1. Add the Windows section, the install.ps1 release asset to the asset table, and a PowerShell verify-by-hand note. Closestracebloc/backend#957 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(supply-chain): correct the raw-<TAG> Windows bootstrap (fails closed w/o $env:REF) Arturo's review (client#336): the §7 parenthetical told customers they could bootstrap install.ps1 from a pinned <TAG> raw URL, but the committed tag tree still ships the __TRACEBLOC_RELEASE_REF__ placeholder (only the release asset is stamped), so that path refuses to run unless $env:REF is set. Documented the $env:REF requirement + why, instead of implying the raw <TAG> URL works as-is — aligned with §1's $env:REF description and the bash side (which uses raw-<TAG> only for hand-verifying a sub-script against the manifest, not for bootstrap). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * feat(installer): first-time install run-through (1. Downloading / 2. Installing a–f) + healthy re-run bailout (#341) * feat(installer): first-run banner/roadmap + step_header + count_bar helpers - print_banner: new title 'Setting up tracebloc on your machine · <version>' (tracebloc bold-cyan) + rule; TB_VERSION from TRACEBLOC_INSTALL_REF; skips when the bootstrap already drew it (TRACEBLOC_BANNER_SHOWN). - print_roadmap: the '2. Installing' a–f plan. - step_header: bold gerund running headers for steps a–f. - count_bar: honest N-of-M render helper for multi-image pulls. - preflight_sudo: step-b password intro copy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): bootstrap early-bailout + "1. Downloading" run-through - Early bailout: if the tracebloc CLI is present and `tracebloc doctor` reports healthy (bounded, exit-code gated), print the healthy line and exec the home screen — skipping the download entirely. Skipped on --force/--reinstall, TRACEBLOC_FORCE_REINSTALL, the dev/unverified path, or an explicit REF/BRANCH. - Draw the first-run banner here and export TRACEBLOC_BANNER_SHOWN (so install-k8s.sh does not draw a second) + TRACEBLOC_INSTALL_REF. - "1. Downloading" copy: Installer downloaded — N files / Verifying it's authentic (cosign)… / Signature verified / All N files intact — nothing was altered. Local colour palette (common.sh not yet sourced). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): step a — collapsed hardware summary + connectivity spinner - run_preflight renders the run-through's step-a view: one hardware line ('arch · N CPU cores · N GB memory · N GB free disk'), a connectivity spinner + combined 'Connected: …' line, and a 'Local storage (~/.tracebloc)' line. - PF_QUIET_SUCCESS suppresses the per-check ✔ lines only inside run_preflight (folded into the summary); called directly (bats), the checks still print their ✔/info lines, so the unit contracts hold. Warnings + hard-fails always print. - Connectivity probes stay in the foreground (PF_HARD_FAIL propagation) with a no-sleep per-host spinner frame. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): step b — Docker %-bar copy + 'ready' success lines - Real %-by-bytes bar for the Docker Desktop .dmg (single-file curl via download_with_progress); fresh-Mac intro copy on the label. - 'Docker ready' and 'System tools ready (k3d, helm, kubectl)' to match the run-through. Linux (Docker Engine) copy left untouched — a deliberate follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): step c — spinner around the (silent) k3d create - _create_new_cluster wraps the 1-2 min k3d create in a spinner ('Creating your secure environment…'), the real fix for the long silent gap; spin() waits for the backgrounded create so exit-code capture + proxy-config cleanup are intact. - Runtime intro copy; terminology 'compute environment' → 'secure environment'. - _wait_for_api owns the single 'Secure environment ready' (API confirmed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): step d — device-flow sign-in copy + "Registered as" - Reworked sign-in copy: "open the link on this or any device and enter the code" (print-only; the CLI prints the URL/code/wait — no auto-open claim). - Success line → Registered as "<slug>" (the minted namespace = dashboard name). - CLI install call removed (now step b); keeps the has-tracebloc FATAL guard. - KEPT the interim name/location prompt (deployed CLI still hard-requires --name without a TTY, backend#992) + comment: remove when cli#137 ships. - No internal step header (main() prints "d) Registering this machine"). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): step b — "tracebloc CLI ready" on the fully-clean path Fully-clean verdict (usable now + in new terminals) → "tracebloc CLI ready … verified on your PATH", matching the run-through's Docker/System-tools/CLI "ready" pattern. Edge-case lines stay "installed" (installed but not yet usable here). Test assertion updated to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): step e — services count-bar + "tracebloc installed" - Services intro copy (training runner / data manager / live monitor / local database; runs on your machine, data never leaves). - _download_services_progress: honest N-of-M count bar as service images pull (imageID-populated count), bounded + non-fatal; guarded by TB_NO_SERVICE_PROGRESS (set in the bats setup so the mocked-kubectl poll cannot hang). Never a fabricated aggregate %. - Success line "Connected to tracebloc" → "tracebloc installed" (step f/summary owns "Connected"); dropped the internal step 4/5 headers (main owns a–f). - Tests: retarget the 3 assertions + guard the poller. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): step f — rich connected summary matching the run-through - connected summary: "✔ Connected to tracebloc"; Environment/Version/Mode block; "live 🟢" + dashboard; NON-dim "What's next" with 3 numbered steps (tb data ingest / my-use-cases / invite collaborators); prominent "Run tracebloc to get started."; dim footer (Logs · Data) with the reboot tip as the LAST dim line. Dropped the green ━━━ border. - Terminology: "secure environment"; trust claim "never leaves this machine". - _reboot_note stays OS-guarded (Linux: restarts automatically; macOS: open Docker Desktop) — Linux not regressed. - wait_for_client_ready intro reframed as step-f "Connecting…". - Tests updated to the new copy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(installer): main() reorg to the six-step run-through (a–f) + gate slot - main() now: banner → [client#339 gate SLOT] → roadmap → a) Check your machine → b) Install what tracebloc needs → c) Create your secure environment → d) Register this machine → e) Install tracebloc → f) Connect to the tracebloc network, each with a step_header + trailing blank-line pair. - Step b now owns prerequisites AND the tracebloc CLI (moved out of provisioning; step d needs it to sign in). - Gate slot: guarded NO-OP call to assess_existing_install, clearly commented as client#339's — logic NOT implemented here; positioned after banner/before roadmap so it reconciles cleanly with that branch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(installer): regenerate manifest.sha256 for the first-run UX changes Content hashes for the 9 edited sub-scripts (common/preflight/setup-macos/ cluster/provision/install-cli/install-client-helm/summary/install-k8s). No files added or renamed, so install.sh's FILES array is unchanged; gen-manifest.sh --check passes (both bootstrap-sync checks green). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(installer): cover the first-run UX logic (Linux CI is the authority) - common.bats: count_bar (render/clamp/bad-input/divide-by-zero), step_header, print_roadmap (a–f plan), print_banner (version + bootstrap-suppression). - preflight.bats: _pf_hw_summary_line, connectivity combined 'Connected:' line, run_preflight collapsed step-a view (per-check ✔ lines folded away). - install-client-helm.bats: _download_services_progress guards (TB_NO_SERVICE_ PROGRESS / no-kubectl / empty-ns) so the poller can never hang the suite. - install-bootstrap.bats: early bailout — healthy doctor execs home screen (no download), unhealthy does not bail, --force skips the bailout. Not run locally (macOS [[ ]] blindspot + spin/sleep/read block without a TTY); assertions written fail-loud against the exact emitted copy for Linux CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(installer): retarget two assertions to the first-run copy Linux CI caught two stale assertions the local checks couldn't (macOS [[ ]] blindspot): - install-bootstrap happy path grepped the old "installer files verified" line, now "All N files intact — nothing was altered". - _pf_storage_type local-fs test grepped the fstype (ext4), which the first-run redesign moved into the log; the visible line is now the clean "Local storage (…)". Assert that instead. Both tests keep their original intent; no code behavior changed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): address Bugbot #341 — real tracebloc cmd, ~/.local/bin bailout probe, exec fallthrough - summary.sh "What's next" said `tb data ingest`, but there is no `tb` on PATH (the binary is `tracebloc`; the CTA two lines below already says `tracebloc`) → use `tracebloc data ingest`. - install.sh bailout probed `command -v tracebloc` before any PATH prepend, so a healthy CLI in ~/.local/bin (the installer's fallback dir, not on a fresh curl|bash PATH) was missed → forced a needless full re-download. Prepend ~/.local/bin first, mirroring provision_client. - install.sh bailout `exec tracebloc || true; exit 0` silently exited 0 if exec failed (bad interpreter / missing exec bit) — healthy machine, but no home screen and no install. Dropped `|| true`; if exec returns (i.e. it failed), print an actionable line and exit 1. Regenerated scripts/manifest.sha256 (gen-manifest.sh --check passes); retargeted the summary.bats assertion to the corrected command. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): redirect bailout CLI stdin away from the curl|bash pipe (Bugbot #341) Bugbot HIGH (learned rule): under `curl … | bash` the healthy-setup bailout ran `tracebloc doctor` and `exec tracebloc` without redirecting stdin, so the child CLI inherited the install pipe as stdin — a CLI that reads stdin could block the bailout or consume the pipe instead of behaving non-interactively. - `tracebloc doctor` (bounded health probe): stdin ← /dev/null (non-interactive). - `exec tracebloc` (interactive home-screen hand-off): stdin ← /dev/tty (the user's real terminal, not the pipe); the exec-failure fallthrough (incl. no controlling terminal to open /dev/tty) keeps the existing "couldn't launch → exit 1" path. bash -n + shellcheck clean. install.sh is the bootstrap (not in the signed manifest), so no manifest change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): probe /dev/tty openability before the bailout exec (fixes#341 CI) c19b51d's `exec tracebloc </dev/tty` is unconditional, but a readable /dev/tty device node with no controlling terminal (CI, detached sessions) still fails to OPEN — the redirect then aborts the exec, the healthy bailout prints "could not launch" + exit 1, and every job that runs install.sh (bats, unit, prereqs, path-persist) goes red. Probe openability first: `</dev/tty` when it opens, else `</dev/null` — still always execs (never the install pipe), so the hand-off and its bats coverage hold. install.sh isn't a hashed sub-script (manifest unchanged). bats install-bootstrap green; shellcheck clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Merge pull request #344 from tracebloc/test/290-check-drift-cli-invariants test(drift): pin the CLI-assumed chart contract in check-drift.sh * test(e2e-proxy): deflake §4 — retry the squid Service-readiness race + raw log on failure (#346) §4 (the #119 ingestion-egress guard) flaked red on develop (run 29255451968); a re-run of the SAME commit (run 29340987746) passed — a transient, not a #341 regression (#341 never touched this test). Cause: the app pod's section-A curl reaches the in-cluster squid via its Service, but `kubectl rollout status` gates only on the squid pod being Ready, not on its Service endpoints being programmed in kube-proxy. In that window a brand-new pod's first CONNECT to the ClusterIP is refused, and curl gave up before attempting the tunnel (no "Establish HTTP proxy tunnel" line) — so the assertion false-failed. - Add `--retry 5 --retry-connrefused --retry-delay 2` (-m 30) to section A's curl: it rides out the transient connection-refused. A genuine #119 regression still fails all retries, so the guard keeps its teeth. - On the (A) assertion failure, dump the RAW egress-app log + squid pod/endpoints state. This flake surfaced only the filtered B line and a bare "did NOT tunnel"; the raw dump makes any future failure debuggable instead of opaque. bash -n + shellcheck clean. (The k3d e2e can't run locally; CI is the authority.) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): resolve Bugbot findings on #345 (tty handoff, pinned-ref reinstall, jq split) (#347) * fix(installer): resolve Bugbot findings on #345 (tty handoff, pinned-ref reinstall, jq split) Three Cursor Bugbot findings on the develop→main PR: - High — assess.sh `_assess_handoff` ran a bare `tracebloc` with no stdin redirect. Under `curl | bash` that inherits the install pipe, so the interactive home-screen handoff could consume script bytes or block (same class as #341). Redirect </dev/tty when openable, else </dev/null — mirrors the bootstrap hand-off in install.sh; still no `exec` so the EXIT trap runs. - High — install.sh skipped its healthy early-bailout on a pinned REF/BRANCH/unverified request but never told install-k8s.sh's own stop-and-check gate. A pinned-ref re-run would download the new installer and then short-circuit to the home screen, doing nothing. Export TB_FORCE_REINSTALL whenever the bailout is skipped so the downstream gate runs the full flow. - Low — assess.sh `_assess_cluster_servers_running` reintroduced a jq/awk bifurcation; jq is not a guaranteed installer prerequisite (Bugbot #284). Collapse to the single jq-free awk path (the SERVERS column read already there). Adds bootstrap tests asserting the reinstall intent reaches install-k8s.sh; test suites green (assess 31/31, install-bootstrap 15/15). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(supply-chain): regenerate manifest.sha256 for assess.sh change Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): give the healthy-machine hand-off a real terminal on stdout/stderr (#348) Bugbot (Medium) follow-up to #347: main() runs setup_log_file (`exec > >(tee …) 2>&1`) BEFORE the stop-and-check gate, so by the time _assess_handoff runs the shell's stdout/stderr are a pipe to tee, not the terminal. The earlier fix redirected only stdin, so the interactive `tracebloc` home screen still rendered onto the tee pipe instead of a tty (the bootstrap bailout in install.sh avoids this only because it hands off before any tee redirect). Point all three streams at the terminal ($TB_TTY, /dev/tty) when it's openable — bypassing tee for the interactive screen, exactly as the bootstrap does — else fall back to </dev/null and leave stdout/stderr on the pipe (non-interactive/CI). Adopt provision.sh's TB_TTY indirection so the redirect is testable; assess.bats now proves the home screen lands on the terminal (not the pipe) on the openable path and falls back cleanly otherwise. Manifest regenerated. assess.bats 32/32. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): assess gate checks cluster is running before the Helm probe (#349) * fix(installer): check cluster is running before the Helm probe in the assess gate Bugbot (High) follow-up: _assess_classify called detect_installed_client (unbounded `helm list -A` / `helm get values`) BEFORE the cluster-servers-running check. On a stopped cluster (post-reboot or manual stop) the k8s API is down, so Helm hangs — violating the gate's "bounded, never-hang" contract — and, when it finally fails, the machine is mislabeled fresh/cluster-no-release, making the cluster-stopped path effectively unreachable on real re-runs. Reorder so the cheap read-only k3d servers-running probe runs first: a stopped cluster short-circuits to degraded/cluster-stopped without ever touching Helm, and detect_installed_client only ever runs against a live API. Adds an ordering guard test asserting the Helm probe is not invoked on a stopped cluster. Manifest regenerated. assess.bats 33/33. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(e2e-proxy): deflake §4 — retry the app-pod proxy-DNS resolution race The §4 app-pod curl used --retry-connrefused, which retries a refused CONNECT (Service endpoints not yet programmed) but NOT a DNS failure. Under CI load the squid Service name isn't yet in the new pod's resolver when curl runs, so it fails with "Could not resolve proxy" (curl exit 5) — which curl does not retry on its own — and §4 flaked red (PR #349 run 29345383969) even though the squid pod and endpoints were up. Add --retry-all-errors (covers the DNS-resolution error too) and bump to --retry 8. A genuine #119 regression still fails all retries, so the guard keeps its teeth. Test-only; not part of the installer supply-chain manifest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): fail closed on unreadable one-client guard values; fix bootstrap spinner on bash 3.2 (#350) Two Bugbot findings on #345: - Medium (install-k8s.ps1): the Windows one-client-per-machine guard treated an unreadable `helm get values` (fetch failure OR unparsable JSON) for a client-chart release the same as "no client here" — so if the ONLY installed client's values couldn't be read, $existingId stayed empty and the install proceeded, silently overwriting a client it couldn't identify (fail-open). Now record any client release whose clientId we can't read and fail CLOSED: refuse with an actionable message rather than overwrite an unknown client. A parsed release with no clientId (literal `null`) is unchanged — still not a match. Adds Pester coverage for both unreadable paths. - Low (install.sh): the early-bailout spinner stored 3-byte braille frames in a single string and indexed `${frames:i:1}` / `${#frames}` — byte-based on macOS system bash 3.2, so glyphs sliced mid-byte and rendered as garbage. Switch to an array of glyphs indexed by element (matches spin() in lib/common.sh), which is char-correct on 3.2. Manifest regenerated (install-k8s.ps1 hash). install.sh shellcheck clean, .ps1 parses cleanly. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): bound the services progress-bar kubectl calls with --request-timeout Bugbot (Medium): _download_services_progress claims to be bounded/non-blocking, but its two `kubectl get pods` calls omitted --request-timeout. The TB_PULL_TIMEOUT deadline is only checked BETWEEN iterations, so a wedged/ unreachable API makes kubectl block indefinitely — freezing step e's progress bar before the authoritative readiness gate in step f ever runs. Add --request-timeout (default 5s, overridable via TB_PROGRESS_KUBECTL_TIMEOUT) to both calls, mirroring assess.sh's bounded probe — a call that can't reach the API now returns quickly, the loop re-checks the deadline, and step e degrades to the honest "still downloading in the background" line instead of hanging. Manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): proactive hardening sweep (bounded probes, tty handoffs, fail-closed guards) (#352) * fix(installer): bound the services progress-bar kubectl calls with --request-timeout Bugbot (Medium): _download_services_progress claims to be bounded/non-blocking, but its two `kubectl get pods` calls omitted --request-timeout. The TB_PULL_TIMEOUT deadline is only checked BETWEEN iterations, so a wedged/ unreachable API makes kubectl block indefinitely — freezing step e's progress bar before the authoritative readiness gate in step f ever runs. Add --request-timeout (default 5s, overridable via TB_PROGRESS_KUBECTL_TIMEOUT) to both calls, mirroring assess.sh's bounded probe — a call that can't reach the API now returns quickly, the loop re-checks the deadline, and step e degrades to the honest "still downloading in the background" line instead of hanging. Manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): proactive hardening sweep — bounded probes, tty handoffs, fail-closed guards A comprehensive audit of the installer (bash + PowerShell) for the failure classes Bugbot has been surfacing one at a time on #345, fixing the genuine instances in one pass. Bounded probes (no more indefinite hangs against a wedged/unreachable API): - cluster.sh: `kubectl cluster-info` in the _wait_for_api gate (the 60s cap was only re-checked between iterations — a single call could block forever). - summary.sh: cluster-info / get nodes / get pods / logs on the summary + not-ready diagnostic paths. - diagnose.sh: every kubectl call in the support-bundle path (`set +e` stops aborts but NOT hangs); helm calls now run only behind a bounded cluster-info probe (helm has no --request-timeout). - gpu-plugins.sh: `kubectl get nodes` in the verify_gpu poll loop. - common.sh download_with_progress: HEAD probe (-m) + the backgrounded curl (--connect-timeout + --speed-limit/--speed-time stall abort — it was monitored only by `kill -0`, no deadline, no kill). - install.sh: every bootstrap fetch (--connect-timeout/--max-time; retry already present, so a stall becomes retriable). - install-cli.sh: the CLI-installer download (a stall now falls to "install later" instead of hanging the step). curl|bash tty handoffs (stdin is the install pipe; stdout/stderr are the tee): - provision.sh: `tracebloc login` (the credential-mint device flow) now gets the real terminal on all three streams when openable, else </dev/null — same idiom as assess.sh's hand-off. - gpu-nvidia.sh: the reboot `read` read from the EOF pipe with no `|| true`, aborting the whole installer under `set -e` right after a successful driver install; now reads /dev/tty, no-tty => no reboot. - setup-macos.sh: the Docker-arch replace prompt read the pipe (empty answer => meaningless confirm); now reads /dev/tty. Fail-closed guards (a failed check must not read as "safe to proceed"): - install-client-helm.sh detect_installed_client now reports INSTALLED_CLIENT_UNKNOWN=1 when `helm list` FAILS (vs genuinely no releases), and the one-client guard refuses rather than risk overwriting a client it couldn't enumerate. +bats coverage. - install-k8s.ps1 one-client guard: same fix at the `helm list` level (non-zero exit OR non-JSON now fails closed, matching the per-release fix from #350). +Pester coverage. set -e footgun: - common.sh _chart_version: trailing `|| true` so a no-match `grep` (no client release) can't abort callers that assign it under `set -e`. PowerShell 5.1 portability: - install-k8s.ps1: the WSL-path build used a scriptblock `-replace` (PS 6.1+); under Windows PowerShell 5.1 (the bootstrap target) the drive letter wasn't lowercased -> malformed path -> 180s NCT-install timeout. Now -match/$Matches. Deliberately NOT changed (documented): - k3d `cluster create --wait` has no --timeout in EITHER bash or PowerShell — at parity, and the known --wait hang cause (proxy misroute) is already mitigated by the proxy config. Not changing critical-path create behavior speculatively. - The dataset-mount check's inspect-failure no-op is a documented, tested design choice (cluster.bats) — left as-is. - Long tail of Linux-GPU-only driver-download curls (gpu-nvidia/gpu-amd, setup-linux) and misc PowerShell -TimeoutSec: LOW severity, retry-wrapped; left for a follow-up to keep this reviewable. All bash suites green (assess 33, install-client-helm 54, cluster 27, provision, summary, install-cli, bootstrap, preflight); shellcheck --severity=error clean; install-k8s.ps1 parses cleanly; manifest regenerated. Two pre-existing local-env bats failures (_extract_yaml_value '' escape; validate_config) are unrelated and also fail on develop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): fail closed on unknown helm state in the #303 pre-provision guard Bugbot (High) on the sweep: the sweep taught detect_installed_client to signal INSTALLED_CLIENT_UNKNOWN=1 on a failed `helm list` and wired it into the Helm-step one-client guard — but provision_client's #303 pre-flight still only checked INSTALLED_CLIENT_NS. A failed enumeration leaves both globals empty, so provisioning continued to `client create` and could register a dashboard client the later Helm guard then refuses to install — the exact orphan the pre-flight exists to prevent. Fail closed on INSTALLED_CLIENT_UNKNOWN right after detect_installed_client, before any mint — same signal the Helm-step guard keys on. +bats coverage (unknown state refuses before `client create`, no orphan). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Merge pull request #353 from tracebloc/fix/bugbot-345-detect-values-failopen fix(installer): bash guard fails closed on unreadable client values (PS parity) --------- Co-authored-by: lukasWuttke <54042461+LukasWodka@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Arturo Peroni <arturo@tracebloc.io>

…ocess (backend#2350) (#811) `E2E auth-proxy (squid)` failed 2 of the last 30 develop runs, both times on the same named assertion: ✖ App pod WITH the ingestion proxy env did NOT tunnel through the squid The probe already carried retries meant to cover exactly the cluster-DNS startup window, under a comment asserting that `--retry-all-errors` covered the "Could not resolve proxy" case. It does not. curl caches a FAILED name resolution for the life of the process, so a single curl's `--retry` re-uses the failure instead of re-querying the resolver. Both failing runs show it — nine attempts, eight of them answered from the cache: * Could not resolve proxy: tb-egress-squid.default.svc.cluster.local * Negative DNS entry curl: (5) Could not resolve proxy: tb-egress-squid... So exactly ONE resolver query was ever issued, about a second after the pod started, and the guard turned entirely on whether CoreDNS happened to be serving at that instant. Confirmed A/B in curlimages/curl:latest — the image the pod runs — with the proxy name made resolvable 4s into the run: the one-process form failed all 9 attempts on the stale negative entry, while the fresh-process loop re-resolved on the very next attempt. 8.20.0 behaves the same, so this was never a floating-tag regression; the claim was wrong from the start. The probe now loops fresh curl PROCESSES, which cannot inherit the poisoned cache, and reports attempt number, exit code and elapsed seconds so a future red says whether it waited — the old failure could not distinguish "did not tunnel" from "had not tunnelled yet". Not a blind retry: only exit 5/6/7 (unresolvable proxy, unresolvable host, refused connection) are retried. Every other outcome, success included, ends the probe, so a real #119 regression — proxy env ignored, so the call dials direct and succeeds with no CONNECT — still fails on attempt 1 rather than being retried into a slow green. A squid that is genuinely down exhausts the deadline and fails. The probe is emitted from e2e-common.sh rather than written inline in the pod manifest so the new bats file executes that same text (backend#1729 rule 9), and it runs in `Unit tests`, which is required on develop. Six tests, each mutation-proved with the anchor asserted: reverting to the in-process retry, retrying every exit code, dropping the fail-closed host guard, removing the give-up report and drifting the manifest indent each redden the specific test named for them. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Add request proxy url to jobs manager yaml file
273835a

saqlainsyed007 requested a review from saadqbal as a code owner May 12, 2026 12:35

saadqbal approved these changes May 12, 2026
View reviewed changes

saqlainsyed007 merged commit af26953 into developMay 12, 2026
4 checks passed

This was referenced May 14, 2026
Config: recognize REQUESTS_PROXY_URL env (peer of jobs-manager networking) tracebloc/data-ingestors#82
Closed
Sync develop → main for v1.3.5 chart release #137
Merged

saadqbal mentioned this pull request May 20, 2026
Add Requests Proxy server to Helm chart #94
Closed

LukasWodka mentioned this pull request Jun 17, 2026
test(e2e-proxy): exercise application-pod egress through the proxy ([redacted] setup) #264
Merged

LukasWodka mentioned this pull request Jul 14, 2026
test(e2e-proxy): deflake §4 — retry the squid Service-readiness race #346
Merged

LukasWodka mentioned this pull request Aug 14, 2026
ci: move checkout and setup-python off Node 20 before the fallback goes #721
Merged

aptracebloc mentioned this pull request Aug 19, 2026
ci: one shellcheck gate, and it is the one that does not apt-install (#753) #754
Merged

LukasWodka mentioned this pull request Aug 24, 2026
fix(e2e): the squid probe's DNS retry was inert — retry in a fresh process (backend#2350) #811
Merged

cursorBot mentioned this pull request Aug 24, 2026
release-train: develop -> staging #815
Merged

Navigation Menu

Uh oh!

Add request proxy url to jobs manager yaml file - #119

Add request proxy url to jobs manager yaml file#119
saqlainsyed007 merged 1 commit into
developfrom
request-proxy

saqlainsyed007 commented May 12, 2026•
edited by cursor Bot
Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

saqlainsyed007 commented May 12, 2026• edited by cursor BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related

Type of change

Test plan

Screenshots / recordings

Deployment notes

Checklist

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

saqlainsyed007 commented May 12, 2026•
edited by cursor Bot
Loading