Runner rejects all backend requests with 401 after v0.2.2 upgrade

[jeremyeder]

Problem

After the v0.2.2 release, all runner pods reject requests from the backend-api and operator with 401 Unauthorized. This breaks session creation, workflow reconciliation, and the between-run listener across all project namespaces on vteam-uat.

The runner starts correctly (AG-UI token authentication enabled), and the operator successfully provisions and refreshes tokens (Refreshed runner token for session ...), but the token the backend sends is not accepted by the runner.

This is an interaction between two PRs that both merged before v0.2.2:

• fix(security): authenticate AG-UI runner endpoints to prevent cross-session attacks #1378 — added X-Ambient-Session-Token header validation middleware to the runner, and runnerTransport in the backend to inject the header
• feat(runner,manifests): alpha migration PR 6+7 — runners and Kustomize overlays #1379 — refactored runner app.py startup, restructured endpoint registration (added events_router), and modified the middleware/lifespan ordering

Likely causes to investigate

• Token secret key mismatch (operator writes to one key, runner reads from another)
• Header name mismatch between what runnerTransport sends and what _require_session_token checks
• Middleware registration ordering changed by feat(runner,manifests): alpha migration PR 6+7 — runners and Kustomize overlays #1379's endpoint restructuring
• The backend's getRunnerAGUIToken() reads from a secret that the v0.2.2 operator writes differently than v0.2.0

Reproduction

Deploy v0.2.2 to a kind cluster and attempt to create a session. The runner pod logs will show 401 Unauthorized on every inbound request.

Affected files

• components/runners/ambient-runner/ambient_runner/app.py (auth middleware)
• components/backend/handlers/sessions.go (runnerTransport, getRunnerAGUIToken)
• components/operator/internal/handlers/sessions.go (regenerateRunnerToken)

Context

Discovered during investigation of vteam-uat outage on 2026-04-22. This is the user-facing symptom — sessions fail with "Error: Runner error: HTTP 401". Affected namespaces include test-jumpstarter-bugs, cve-fixer-workflow, maas-cve-fixer, chocolate-factory, and others.

[github-actions]

Inactive for 30 days. Will close in 7 days unless there's activity.