fix(egress-enforcement-check): retry probe to tolerate standard-mode CNI reconciliation window

[saadqbal]

What

Fixes the SECURITY §8.2 egress-lockdown enforcement helm test (egress-enforcement-check) false-negative on standard-mode CNI clusters — notably AWS VPC CNI with NETWORK_POLICY_ENFORCING_MODE=standard.

Closes#275. Follow-up to tracebloc/client-runtime#104; unblocks the per-fleet rollout gate for tracebloc/client-runtime#102.

Why

standard mode allows a brand-new pod all traffic until its per-pod policy reconciles (a few-second startup window). The probe curled the canary host immediately on startup, won that race, connected, and reported EGRESS LOCKDOWN NOT ENFORCED — even though the lockdown is enforced for real training pods (which start far slower than the window). Both dev (tb-client-dev-templates) and prod (tracebloc-clients-prod) run AWS VPC CNI in standard mode, so as-is the gate would false-fail on every fleet and could mislead an operator into thinking the lockdown is broken.

Change

• Retry-until-blocked — the probe loops until egress is observed blocked (curl exit 7/28 → PASS), bounded by a new networkPolicy.training.enforcementProbeTimeoutSeconds (default 60s); FAILs only if the host stays reachable for the whole timeout. Correct on standard mode, strict mode, and genuinely non-enforcing clusters alike.
• Docs — refreshed the stale values.yaml CNI guidance: AWS VPC CNI does enforce with enableNetworkPolicy=true (the prior comment claimed it doesn't).
• Tests — +2 helm-unittest cases (retry loop honours the configurable timeout; default 60s). Full chart suite: 267/267.

Verified on dev (EKS, 2026-06-24)

Applied the lockdown on tb-client-dev-templates and probed from a tracebloc.io/workload=training pod:

• helm test (old single-shot behaviour) → Failed (probe connected on startup).
• a training-labelled pod that sleeps 30s then probes → 1.1.1.1:443 and example.com:443blocked (curl exit 28); backend reachable via the squid gateway (exit 0).

So the lockdown works for real pods; only the instant probe raced it. Dev was rolled back to its original state after verification.

Not in scope

standard mode's startup window is itself a narrow egress gap for untrusted pod code; strict mode closes it but is a cluster-wide change — tracked separately as a §8.2 threat-model decision.

🤖 Generated with Claude Code

---

Note

Low Risk
Changes only the helm test hook script, values docs, and chart tests; no runtime training or NetworkPolicy behavior is modified.

Overview
Fixes false FAIL results on the §8.2 egress-lockdown helm test when the CNI enforces policy only after a short per-pod reconcile (e.g. AWS VPC CNI standard mode).

The egress-enforcement-check probe no longer curls once at startup. It loops (3s between attempts) until curl sees blocked egress (exit 7/28) → PASS, or until networkPolicy.training.enforcementProbeTimeoutSeconds (default 60s) expires while the host stayed reachable → FAIL. A reached latch records any successful TCP connect so a later DNS failure cannot flip a real non-enforcement case to “inconclusive.”

values.yaml documents the new timeout, updates EKS guidance (VPC CNI can enforce with enableNetworkPolicy=true), and refreshes failure-message hints. Helm unittest adds coverage for the retry loop, timeout wiring, and the 60s default.

^{Reviewed by Cursor Bugbot for commit 5ba4646. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit db87393. Configure here.}

[saadqbal]

/fr-pass

Functional review passed on dev: the patched egress-enforcement-check helm test goes green on a real standard-mode VPC CNI cluster (tb-client-dev-templates), and a live egress-restriction demo confirmed a training pod's direct external calls are blocked while only the allowlisted backend is reachable via the squid gateway.