feat(egress-proxy): training-pod egress lockdown — squid gateway, gated rollout (client-runtime#102)

[saadqbal]

Part of tracebloc/client-runtime#102 (SECURITY §8.2 — training-pod egress lockdown). Chart side; companion PR: tracebloc/client-runtime (jobs-manager HTTPS_PROXY injection).

Why

A training pod runs untrusted, externally-uploaded ML code, may read the dataset from in-cluster MySQL, and can today open outbound HTTPS to 0.0.0.0/0:443 (training NetworkPolicy rule 2) — a data-exfiltration path that undercuts the "data never leaves your infra" guarantee.

What

Everything ships inert behind flags — no fleet behaviour changes on upgrade until an operator opts in per fleet:

1. Egress gateway (egressProxy.*, default deployed but unused): squid forward proxy, HTTPS CONNECT only to an FQDN allowlist (backend per env + App Insights; fail-closed), chains to a corporate proxy via cache_peer (mirrors tracebloc.proxyEnv inputs). app=egress-proxy (deliberately outside the training netpol), ClusterIP egress-proxy-service:3128, PSA-restricted, read-only rootfs, pid_filename none, pinned to the multi-arch (amd64+arm64) ubuntu/squid index digest.
2. Routing (egressProxy.routeWorkloads, default false): jobs-manager gets EGRESS_PROXY_URL and injects HTTPS_PROXY into training pods (companion PR); netpol gains a training→gateway:3128 egress rule.
3. The lockdown (networkPolicy.training.allowExternalHttps, default true): when explicitly false, rule 2 (0.0.0.0/0:443) is dropped — training pods then reach only DNS, MySQL:3306, requests-proxy:8888, and the gateway:3128. Implemented with dig … true, so an absent key (stored-values replay) keeps the rule: the safe direction.
4. Docs: SECURITY.md §4.2/§8.1/§8.2 rewritten to "mechanism shipped, gated rollout" with the per-fleet procedure. BACKEND_TOKEN residual stays tracked under §8.1.
5. CI: new upgrade-e2e job — installs the last published chart on a real k3d cluster, upgrades to this tree via both--reuse-values (nil-guards hold, lockdown must not engage by accident) and --reset-then-reuse-values (the auto-upgrade cronjob's path: new defaults flow in inert, image-refresh annotations survive), then flips the lockdown flags and proves the next auto-upgrade preserves the flip.

Auto-upgrade safety

• The hourly auto-upgrade (--reset-then-reuse-values) carries the new defaults through and preserves operator overrides — flips are durable.
• All new keys nil-guarded for manual helm upgrade --reuse-values from pre-1.7.0 stored values (regression-tested in egress_proxy_test.yaml + the e2e gate).
• helm-unittest pins the training netpol podSelector to exactly tracebloc.io/workload: training, so the lockdown can never catch the auto-upgrade/image-refresh cronjobs.

Rollout (per fleet, each step reversible)

1. Upgrade to 1.7.0 → gateway deploys, inert.
2. egressProxy.routeWorkloads=true → verify a training run completes via the gateway.
3. networkPolicy.training.allowExternalHttps=false → verify G2: in-pod curl https://example.com fails, run still completes. Requires a NetworkPolicy-enforcing CNI (Calico/Cilium/OVN — not flannel/kindnet).

Tests

• helm unittest: 212/212 (20 suites) — incl. --reuse-values replay, flag gating both ways, podSelector stability.
• helm lint + full helm template clean; chart 1.6.1 → 1.7.0 (version+appVersion lockstep).
• Squid smoke-tested in docker under prod constraints (read-only rootfs, uid 13, pinned digest): backend 200 through the allowlist, example.com denied.
• bash -n + shellcheck --severity=error clean on the new e2e script.

Remaining before fleet rollout

Cluster G2 e2e on a NetworkPolicy-enforcing CNI (tracked on tracebloc/client-runtime#102).

🤖 Generated with Claude Code

---

Note

High Risk
Changes training egress NetworkPolicy semantics and fleet upgrade paths; misconfiguration or broken nil-guards could break training or accidentally enable lockdown on upgrade.

Overview
Chart 1.6.1 → 1.7.0 adds a gated training-pod egress lockdown (SECURITY §8.2 / client-runtime#102): an optional in-cluster squid egress gateway (egressProxy.*), training NetworkPolicy changes, and jobs-manager wiring for EGRESS_PROXY_URL when routing is enabled.

Egress gateway (default on, inert until opted in): New ConfigMap/Deployment/Service render a fail-closed FQDN allowlist proxy (app=egress-proxy, outside the training netpol). egressProxy.routeWorkloads defaults false; when true, jobs-manager gets EGRESS_PROXY_URL for companion runtime HTTPS_PROXY injection.

Network lockdown: Rule 2 (0.0.0.0/0:443) is wrapped in dig "allowExternalHttps" true so missing keys on --reuse-values keep legacy behavior; explicit false drops direct external HTTPS. When the gateway is enabled, training pods get an explicit egress rule to egress-proxy on the proxy port.

Safety & CI:values.yaml, values.schema.json, helm-unittest suite egress_proxy_test.yaml, and SECURITY.md document the progressive rollout. New upgrade-e2e job runs scripts/tests/e2e-auto-upgrade.sh on k3d (published chart → local tree via --reuse-values and --reset-then-reuse-values, then lockdown flag flip persistence). Shellcheck/bash gates include the new script.

^{Reviewed by Cursor Bugbot for commit da57914. Bugbot is set up for automated code reviews on this repo. Configure here.}