Add authenticated WebSocket recovery telemetry

[ravarora2]

Summary

Argo rollout recovery needs to distinguish an open WebSocket from a connection that completed NIP-42 authentication and is usable. The relay currently exposes total WebSocket connections, but it cannot answer whether authenticated clients recovered after a pod drain or why authentication failed.

This change adds a bounded recovery contract:

• buzz_auth_attempts_total{method="nip42"}
• buzz_auth_outcomes_total{method="nip42",outcome}
• buzz_auth_duration_seconds{method="nip42",outcome}
• buzz_ws_authenticated_connections_active
• buzz_auth_post_terminal_frames_total{state}

buzz_auth_attempts_total now has one consistent recovery-oriented unit: a challenge lifecycle successfully queued to the connection writer. This intentionally replaces the historical unit of AUTH frames that reached the pending handler. AUTH frames received after a lifecycle is already authenticated or failed are protocol noise; they use the separate bounded post-terminal counter and cannot inflate rollout-gating attempts or outcomes.

Each issued challenge reaches exactly one terminal outcome. Authentication transitions and active-gauge changes share one short synchronous state lock, cancellation can claim a pending database-backed AUTH before a late handler result, and a drop guard reconciles aborted or panicked connection futures. Accounting is terminalized before writer joins, while terminal socket delivery has a one-second best-effort bound so a stalled sink cannot retain the authenticated gauge or connection permit indefinitely.

Dependency failures are distinct from policy verdicts: allowlist and relay-membership lookup errors now emit allowlist_check_error and relay_membership_check_error, never allowlist_denied or not_relay_member. The fail-closed allow/deny policy is unchanged. All rollout-facing labels come from fixed enums; identities, challenges, URLs, and raw error strings are never exported.

All fixed counter and histogram label sets, including zero histogram buckets, counts, and sums, are present on the first real exporter scrape. This is the first implementation slice of the rollout-reliability plan. A later PR will consume these metrics from opt-in Argo Rollout analysis; merging this PR alone does not change deployment behavior.

Related issue

None found.

Testing

Validated at final head f9992cac835284c25ee0f99e17ddba0b932d7c69, which contains current origin/main:

• cargo fmt --all -- --check
• cargo clippy -p buzz-relay --all-targets -- -D warnings
• compiled every buzz-relay test target
• release builds for buzz-relay, buzz, buzz-admin, and buzz-test-cli
• 16/16 connection lifecycle/writer/production-dispatch tests
• 7/7 AUTH handler tests
• bounded metric-contract exporter test
• 13/13 relay main-target tests (1 PostgreSQL-only test ignored)
• 9/9 real boot-lifecycle tests, including a production install and first Prometheus scrape

Mutation checks proved that the production regressions fail when any of these are removed or reintroduced:

• challenge attempt start in handle_active_connection
• authentication series initialization in the real metric installer
• separation of post-terminal AUTH floods from authoritative attempts

The exact-head release binaries were also exercised against the isolated review database:

1. Started the release relay and verified health/readiness.
2. Created a channel with the regular CLI.
3. Held one authenticated WebSocket subscription open and authenticated a second WebSocket publisher.
4. Published a message and read it back through the regular CLI.
5. Observed attempt_delta=2, success_delta=2, active_while_subscribed=1, and active_after_disconnect=0.
6. Sent SIGTERM and waited for graceful relay shutdown.

Local caveat

The full relay library target consistently reached 1,049 passes with 89 intentionally ignored tests, but the untouched api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo intermittently returned HTTP 504 instead of 200. One isolated rerun passed; later isolated and full reruns reproduced the 504. No mesh files are changed by this PR, and all changed-path, main-target, boot-lifecycle, lint, build, mutation, and live-local checks above passed at the final head.

Generated with Codex

[github-actions]

🔐 Codex Security Review

Status: review required for the current range.

The current range is 051c3a270be9c73da9ab06700bcab7d5552fceaa...f9992cac835284c25ee0f99e17ddba0b932d7c69.
A new review must complete for this exact range. When manual authorization
is required, a Block organization member must comment exactly
@buzz-security-review f9992cac835284c25ee0f99e17ddba0b932d7c69 to authorize a new review.
Any previous review applies only to its recorded range.