fix(chart): move the prod ingestor pin into chart defaults so it reaches edges

[LukasWodka]

Summary

The prod ingestor digest pin reached no customer edge. This moves it from an
install-time -f overlay into the chart's default values.yaml, gated to prod,
so it propagates through the auto-upgrade the fleet already runs.

Two independent reasons the overlay never applied:

1. Standard installs never passed it. The installer runs
   helm upgrade --install … --values "$values_file" with only its generated
   values file; no installer path ever passed -f client/values-prod.yaml. A
   normal prod install kept the base default images.ingestor.digest: "" and
   floated on tag 0.7 with imagePullPolicy: Always — exactly like dev/staging.
2. Where applied by hand it could never be updated. The auto-upgrade CronJob
   runs helm upgrade --reset-then-reuse-values with no -f and no --set.
   That resets to the new chart's values.yaml defaults, then re-applies the
   release's stored user-supplied values. An overlay value is user-supplied, so
   it was replayed verbatim forever — and Helm only auto-reads values.yaml from
   a chart, so the updated overlay inside the new chart archive was never read.

Chart defaults propagate; user-supplied values freeze. So the pin has to be a
default — which is how the egress-proxy squid image has always been pinned, and
why that one does reach installed edges.

What changed

Key	Kind	Role
images.ingestor.prodDigest	new, chart default	The prod pin. A default, so a republished pin reaches installed edges.
images.ingestor.prodPin	new, default true	Gates the pin. false floats a canary prod edge.
images.ingestor.digest	existing, unchanged semantics	Explicit per-edge pin; wins in any environment.

A new tracebloc.ingestorDigest helper resolves one effective digest:

1. images.ingestor.digest non-empty → that digest, any environment.
2. else prodPin (default true when the key is absent) and resolved
   CLIENT_ENV == "prod" → prodDigest.
3. else empty → float on tag.

Gating mechanism — why CLIENT_ENV. dev/staging installs carry
env.CLIENT_ENV: dev|stg in their user-supplied values, while the template
defaults CLIENT_ENV to prod. So a template-level gate on the resolved
CLIENT_ENV gives prod-pinned / non-prod-floating with zero per-edge action
and no new install-time flag. prodPin defaults to true when the key is absent
so a --reuse-values upgrade from a release predating it doesn't silently
defeat the pin.

The metadata-backfill hook now shares the same helper, so it can never run a
different ingestor build than the ingestions it reconciles.

The consuming contract in client-runtime _build_image_reference is unchanged:
digest set → repo@digest + IfNotPresent; digest empty → repo:tag + Always.

values-prod.yaml — removed, not shimmed

Passing -f values-prod.yaml is precisely what created the frozen-value bug, so
keeping the file as a no-op shim would invite operators to re-create it. It is
deleted and every reference updated. MIGRATION.md documents the move, the
canary knob, and — importantly — how to clear a hand-layered stored
images.ingestor.digest, which still takes precedence and would otherwise keep
such an edge frozen:

helm upgrade <release> tracebloc/client -n <ns> \
--reset-then-reuse-values --set images.ingestor.digest=""

Other

• resolve-ingestor-digest.sh keeps its resolution mechanism (multi-arch index
  digest + guard); only the destination changes. It now writes prodDigest in
  client/values.yaml, keyed on that unique name with a single-match assertion
  so it can never rewrite a sibling image's digest: leaf (values.yaml has one
  per image), and it reminds the operator to bump Chart.yaml.
• CI's ingestor-multiarch guard now validates prodDigest too. It previously
  read only values.yaml, so it never checked the overlay it was meant to guard
  — a gap the old overlay's own comments admitted. It also hard-fails an empty
  pin, which would silently un-pin prod.
• Chart bumped 1.9.5 → 1.9.6 (version + appVersion). The chart only publishes
  on a version change and nothing in CI guards a forgotten bump.
• prodDigest is set to the multi-arch index the published 0.7 tag currently
  resolves to, so switching the pin on is behaviour-neutral on day one: prod
  keeps running exactly the image it runs today, now reproducibly. (The digest
  previously committed in the overlay had already gone stale — see below.)

Test results

• helm unittest ./client — 304 passed, 27 suites (was 297). Added
  helm-unittest cases asserting prod renders the pinned digest and dev/staging
  render the floating tag, plus explicit-CLIENT_ENV=prod, prodPin: false,
  empty-prodDigest fallback, and explicit-digest-overrides-prod-pin.
• helm lint --strict ./client -f client/ci/{aks,bm,eks,oc}-values.yaml — clean
  on all four; helm lint --strict ./ingestor clean.
• helm template proving the gate:
  • no CLIENT_ENV (what the standard prod installer generates) →
    INGESTOR_IMAGE_DIGEST: "sha256:9098b3c9…"
  • --set env.CLIENT_ENV=dev → INGESTOR_IMAGE_DIGEST: ""
• Schema guards: a tag-shaped prodDigest and a non-boolean prodPin are both
  rejected by values.schema.json.
• shellcheck --severity=error clean on both changed shell scripts (warnings
  clean too); bash -n clean across scripts/; scripts/check-style.sh clean;
  scripts/gen-manifest.sh --check reports the manifest current (neither changed
  script is part of the bootstrap's integrity surface, so no manifest change).
• Live cluster verification (throwaway k3d, published 1.9.5 → this tree):
  • Installing the published chart renders an empty digest — prod floats
    today, confirming the bug.
  • --reset-then-reuse-values (the exact fleet command, no -f, no --set)
    lands the chart-default pin on the already-installed edge. Setting
    env.CLIENT_ENV=dev on the same live release floats it again.
  • Counterfactual: the same upgrade leaves an overlay-style user-supplied
    pin frozen on its install-day value, with the chart's newer pin ignored.
    Clearing it with --set images.ingestor.digest="" restores tracking.
  • scripts/tests/e2e-auto-upgrade.sh now asserts all of this permanently,
    including that a prodPin: false opt-out survives the next auto-upgrade.

Note for RFC 0005 / epic tracebloc/backend#1246

The same distribution flaw applies to the training-image digest map proposed
in RFC 0005. A digest map delivered as an install-time overlay would reach no
edge and could never be updated, for exactly the reasons above. That is why the
RFC now specifies chart defaults rather than an overlay — and any per-env
gating there should follow the resolved-CLIENT_ENV pattern established here.

One thing that contradicted the issue

The issue lists the pin as sha256:78f21a08… on main and sha256:a5f2650b… on
develop. Resolving the published 0.7 tag against the registry today returns
neither — it is now sha256:9098b3c9…. The 0.7 line moved again after the
overlay's last # VERIFIED refresh, so that digest was already stale on top of
being undeliverable. Pinning the current head keeps activation behaviour-neutral;
pinning a5f2650b… would have moved prod backwards from what it runs today.

Fixes tracebloc/backend#1245

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e381555. Configure here.}

[divyasinghds]

Approve. The most consequential of the batch; the core reasoning (chart defaults propagate through --reset-then-reuse-values, user-supplied values freeze) is correct and the helper logic holds.

• tracebloc.ingestorDigest precedence is right: explicit digest → else prodPin (default-true) && CLIENT_ENV==prod → prodDigest → else empty/float. Every read is nil-guarded, so a --reuse-values upgrade from a pre-key release still renders (and correctly does not pin — the documented path-1 limitation, asserted in e2e).
• Complete consumer coverage — jobs-manager and metadata-backfill are the only two ingestor-digest consumers; image-refresh-cronjob.yaml explicitly excludes the ingestor, so nothing bypasses the shared helper.
• No dangling values-prod.yaml references after deletion (no installer/CI path ever passed it — which was the bug).
• CI ingestor-multiarch now hard-fails an empty/single-arch prodDigest; resolve-ingestor-digest.sh is hardened with a single-match assertion so it can't rewrite a sibling image's digest: leaf. Chart bumped 1.9.5 → 1.9.6 (required to publish).

The gate rests on the invariant that dev/staging carry env.CLIENT_ENV in user-supplied values (else they'd resolve to prod and pin) — an existing installer guarantee, now locked by the new helm-unittest + e2e cases.

Reviewed with Claude Code.

[aptracebloc]

Post-merge review — for the record, this would have been an approve.

Correct root-cause diagnosis: the prod ingestor pin reached no edge because it lived in an install-time -f values-prod.yaml overlay, and (1) no installer path passed -f values-prod.yaml, while (2) the fleet auto-upgrade's helm upgrade --reset-then-reuse-values resets to the new chart's values.yaml defaults then replays the release's stored user-supplied values verbatim — so an overlay freezes forever and the updated overlay in the new chart archive is never read. Moving the pin into the chart default, gated via one helper, is exactly right: chart defaults propagate through --reset-then-reuse-values; user-supplied values freeze. (Same mechanic as the squid image, which is why that one does reach edges.)

The tracebloc.ingestorDigest helper is well-built:

1. explicit images.ingestor.digest → that digest, any env;
2. else prodPin AND CLIENT_ENV == "prod" → prodDigest;
3. else empty → float on tag.

Two things I especially liked: prodPin defaults to TRUE when the key is absent ($prodPin := true; if hasKey … {$prodPin = …}), so a --reuse-values upgrade from a release predating the key still pins prod rather than silently defeating it; and it's the single source every consumer (jobs-manager + metadata-backfill) resolves through, so they can't disagree. nil-guarded throughout, Bugbot clean, CI green, with MIGRATION.md + schema + template tests.

LGTM (retroactively).

— drafted with Claude (Opus 4.8), sent by @aptracebloc