ci(e2e): give E2E auth-proxy (squid) its own always-reporting workflow (backend#2350)

[aptracebloc]

What & why

E2E auth-proxy (squid) went red when client#789 promoted develop→staging on 2026-08-22, and it blocked nothing. This PR does the remaining enablement so the check can actually gate — the "step 4" the ticket calls the last outstanding work.

Closes tracebloc/backend#2350

(The one operational step this PR cannot do — arming the branch-protection flip on staging+main — is a settings action deferred to after a green-watch window; it is documented in full at the end of this description so it is not lost when the issue closes.)

Verdict on "flake vs. Collector regression" — it is a flake, independently confirmed

Two outcomes were on the table: (a) flake / long-standing red, or (b) a real regression from the telemetry Collector payload (#779/#784/#790/#792). It is (a). Evidence I gathered directly from the runs, not from the issue thread:

• The failing assertion is a cluster-DNS startup race, not a proxy defect. In the failing job (run 32589039697, job 97069826651) the node-level image pulls tunnelled fine through squid — TCP_TUNNEL/200 CONNECT auth.docker.io:443 tbuser appears repeatedly. The failure is only the in-cluster app-pod probe, and its own log prints the mechanism:

  app| * Could not resolve proxy: tb-egress-squid.default.svc.cluster.local
  app| * Negative DNS entry
  app| curl: (5) Could not resolve proxy: tb-egress-squid.default.svc.cluster.local
  

  The first attempt does a real lookup; every subsequent attempt prints * Negative DNS entry — curl served all retries from its process-lifetime negative-DNS cache, so --retry-all-errors never re-queried the resolver. One real lookup, ~1s after pod start, decided the whole check.
• History disconfirms the Collector. On develop, the E2E auth-proxy (squid) job is 28 green / 2 red over the 30 installer-tests runs around the event (6.7%), and it has been continuously green since ebb84b13 — the merge of the fix (fix(e2e): the squid probe's DNS retry was inert — retry in a fresh process (backend#2350) #811) on 2026-08-24 — through today. A regression would be a wall of red, not two isolated failures with green on either side. The 19:57 develop run on 2026-08-22 carried more Collector code than the promotion did and was green.

So: not the payload, and not a permanently-red job either — an intermittent false red in a non-required job, the shape that never gets diagnosed because each instance is individually dismissible.

What was already fixed (no code change needed here)

• fix(e2e): the squid probe's DNS retry was inert — retry in a fresh process (backend#2350) #811 made the probe retry in a fresh curl process per attempt (a new process cannot inherit the negative-DNS cache) and report attempts + elapsed. Merged to develop 2026-08-24.
• chore(tests): pin the e2e-proxy probe image, like the one next door (backend#2350) #813 pinned the probe image; squid and nginx are already pinned by digest in e2e-proxy.sh. The "pin external deps before gating" prerequisite is satisfied.

What this PR does — the remaining enablement (step 4)

The check could not block because it was a job inside installer-tests.yaml, which is path-filtered on both push and pull_request. A required check behind a paths filter never reports on a PR that touches nothing under the filter — GitHub then holds the PR at "Expected — waiting for status to be reported" forever. That is the same hazard drift-checks.yaml documents (it once stuck client#651/#657/#660), and the same reason Source-of-truth drift was moved out of installer-tests' old static job.

This PR extracts e2e-proxy into its own e2e-auth-proxy.yaml, following that precedent:

• pull_request has no paths: filter → the context always reports on every PR to main/staging/develop/openshift, so it can be required on staging/main without deadlocking a promotion PR whose batch happens to touch no scripts/ file (backmerge, hotfix lane, a version-bump promotion).
• push keeps its paths filter (pushes aren't gated) and gains staging; the Monday cron is preserved so extraction doesn't drop the job's weekly canary.
• Job name E2E auth-proxy (squid) is unchanged → the required-context name and the release train's by-name check reading both keep working. No duplicate context: the job is removed from installer-tests.yaml (a tombstone comment records where it went and why).

This choice dominates both options weighed in the ticket: no skip-shim (so none of backend#2447's fail-open shape), and no "drop paths → run the full distro matrix on every PR" cost — only a single ubuntu-latest job moves.

The "should it be required?" decision (step 5) — yes, on staging + main; not flipped here

It should gate, on staging + main (the hop this protects), notdevelop (where a blip would tax every PR). But it is a branch-protection setting, not a file, and flipping it now would be wrong on two counts: the context can't be selected until this workflow has run at least once on those branches, and per arm-while-green it should wait out a green-watch window (#811 landed only 2026-08-24). I have not changed any protection setting. Once this merges, has run, and the watch is clean, arm it with:

gh api -X PATCH repos/tracebloc/client/branches/staging/protection/required_status_checks \
--input - <<'JSON'{"strict": false, "checks": [{"context": "E2E auth-proxy (squid)"}]}JSON# then repeat for branches/main/protection/required_status_checks

(Merge the new context into the existing checks list rather than replacing it — read the current set first with gh api repos/tracebloc/client/branches/staging/protection/required_status_checks --jq '.checks'.)

Test evidence

• Both workflows parse (yaml.safe_load); the only uses: is SHA-pinned (quality / action-pins); no .sh changed (quality / shellcheck out of scope); permissions: contents: read preserved.
• installer-tests.yaml jobs after the edit: unit-bash, unit-pester, distro-prereqs, e2e-cluster, path-persist, e2e-journey — e2e-proxy cleanly removed, no needs: referenced it.
• This PR targets develop, which is in the new workflow's pull_request branches with no paths — so E2E auth-proxy (squid) runs on this PR itself: live proof the extracted workflow is green (the probe script is byte-identical to what is already green on develop).

Reviewer note

The class here — "an expensive E2E stuck in installer-tests that can therefore never gate" — has other members (E2E cluster, E2E last-mile journey). This PR deliberately scopes to the auth-proxy check the ticket is about; whether those two also deserve extraction/gating is a separate call.

Ref: tracebloc/backend#2350

---

Note

Low Risk
CI workflow layout only—no installer or test script changes in the diff; branch protection arming is documented as a separate manual step.

Overview
Moves the E2E auth-proxy (squid) job out of path-filtered installer-tests.yaml into a new e2e-auth-proxy.yaml, matching the earlier Source-of-truth drift extraction so the check can become a required status gate on staging/main.

The new workflow keeps the same job name and still runs scripts/tests/e2e-proxy.sh on a single ubuntu-latest runner. pull_request has no paths: filter, so every PR to main, staging, develop, and openshift gets a status report (avoiding “waiting for status” on promotions that don’t touch scripts/). push stays path-scoped under scripts/** and the workflow file, adds staging on push, and retains the Monday 03:00 UTC schedule plus workflow_dispatch.

installer-tests.yaml drops the e2e-proxy job and documents the move in a tombstone comment; no other jobs depended on it.

^{Reviewed by Cursor Bugbot for commit 0872fb3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saadqbal]

The diagnosis is the best part and it's worth restating for anyone skimming: --retry-all-errors retried nine times and asked the resolver once. curl's process-lifetime negative-DNS cache served every subsequent attempt from the first failure — * Negative DNS entry in the log is the tell — so a retry loop that looked like resilience was replaying a single answer taken ~1s after pod start. A retry that cannot observe the thing it's retrying against isn't a retry, and #811 fixing it with a fresh process per attempt is the only shape that works.

Separating node-level from in-cluster egress is what makes the verdict trustworthy: TCP_TUNNEL/200 CONNECT auth.docker.io:443 tbuser proves squid tunnelled fine, so the failure was never the proxy. And the history is a real control rather than a vibe — 28/2 over 30 runs, continuously green since ebb84b13, and the 19:57 develop run carried more Collector code than the promotion did and was green. A regression would be a wall of red; two isolated failures with green either side is a flake. Gathering that from the runs rather than the issue thread is the difference between a verdict and a repetition.

"An intermittent false red in a non-required job — the shape that never gets diagnosed because each instance is individually dismissible" is the sentence I'd put in a retro. That's why this sat since 22 Aug.

The enablement design is the load-bearing bit and it's right. No paths: on pull_request, because a required check behind a paths filter simply never reports on a PR that touches nothing under it — the check sits "Expected" and the PR can never merge. Keeping paths: on push is the correct asymmetry, since pushes aren't gated by required checks and there's no reason to spend the minutes. Both stated at the top of the file rather than left for someone to rediscover.

And the job name is preserved, so the check context stays E2E auth-proxy (squid). That's the third time this week a PR has had to think about required-context stability — backend#2655 left a job name alone for the same reason, and .github#337 was about a cosmetic name: silently renaming a reusable-workflow context. It keeps nearly costing someone a repo-wide block.

Documenting the branch-protection flip in the PR description because the issue closes on merge is the right instinct — that step has no other home once #2350 is shut, and "deferred to a green-watch window" is a decision, not a TODO.

Green, no threads. 👍