fix(dev): restore fallback capacity for canonical E2E chat verification

[sheepdestroyer]

Description

The dev canonical verifier cannot currently complete its three model-dependent checks even though deployment, infrastructure, Langfuse propagation/leak isolation, and all public canonical routes pass.

Actual vs expected

Expected: python3 scripts/verification/verify_canonical_endpoints.py --dev completes every chat route successfully.

Actual (2026-07-23): 26/29 passed. The llm-routing-auto-free simple and medium requests timed out, and a direct LiteLLM agent-simple-core request timed out.

Root cause

The failure is upstream/infrastructure availability rather than the Quadlet or subdomain changes in the replacement PR:

1. OpenRouter responded 429 Rate limit exceeded: free-models-per-day-high-balance, with X-RateLimit-Remaining: 0.
2. After external fallbacks were cooled down, the local safety-net route attempted to contact x570.vendeuvre.lan:443 from inside the dev LiteLLM container and received ConnectionRefusedError.

Failure cascade

Step	Component	Evidence	Effect
1	OpenRouter free roster	LiteLLM logs: HTTP 429 / remaining quota 0	tier deployments enter cooldown
2	LiteLLM fallbacks	agent-simple-core falls through tier groups	request latency expands until verifier timeout
3	local-qwen-3.6 / Ollama safety-net	LiteLLM logs: connection refused to x570.vendeuvre.lan:443	no terminal local fallback is reachable
4	canonical verifier	3 chat checks time out	verifier exits 1 despite 26 passing checks

Affected components

• scripts/verification/verify_canonical_endpoints.py: auto simple/medium and direct LiteLLM chat checks.
• Dev LiteLLM fallback chain and the local safety-net upstream reachability.
• Not affected in this run: router/litellm/langfuse/minio/clickhouse health, Langfuse session propagation + leak isolation, or all router/LiteLLM/Langfuse/llama canonical HTTPS checks.

Environment

• Dev pod: dev-router-pod
• Router/LiteLLM: 127.0.0.1:5010 / 127.0.0.1:4010
• Public routes verified: dev.vendeuvre.lan, litellm.dev.vendeuvre.lan, langfuse.dev.vendeuvre.lan, llama.dev.vendeuvre.lan

Requirements

1. Restore/replace the exhausted OpenRouter free-tier capacity for dev verification.
2. Trace the local safety-net URL from rendered LiteLLM configuration to DNS/HAProxy and make it reachable from the dev container without coupling dev to a broken production hostname.
3. Re-run the full dev canonical verifier and record a clean result.

[github-actions]

The issue involves the dev canonical verifier failing three of 29 chat-related checks due to infrastructure limitations and exhausted upstream resources. OpenRouter's free-tier quota was depleted, leading to rate-limiting, and local fallback mechanisms failed due to unreachable endpoints within the LiteLLM container. While core components and public routes remain functional, the fallback chain requires restoration or adjustment to ensure all verification routes complete as expected. Key actions include addressing resource availability, resolving network reachability issues, and re-running the verification process for confirmation.

[sheepdestroyer]

Update after #347 dev deployment validation:

• Resolved the unreachable safety-net upstream configuration: dev now renders LLAMA_CLASSIFIER_URL=http://127.0.0.1:8083/v1 and no longer sends fallback traffic to x570.vendeuvre.lan:443.
• The local listener is reachable: GET http://127.0.0.1:8083/health and /v1/models returned 200; LiteLLM logs show the local fallback selecting api_base: http://127.0.0.1:8083/v1.
• The remaining local fallback failure is runtime capacity/load contention, not routing reachability: llama-router repeatedly proxies Qwen3.6-27B while waiting for/loading qwen-4b-routing / Qwen3.6-27B-spec3; the direct local-qwen-3.6 completion exceeded 120 seconds. The E2E verifier therefore still has two auto-route timeouts while OpenRouter reports free-tier quota remaining 0.

#347 contains the reachability correction. This issue remains open for restoring deterministic model capacity (or adapting the verification model/load policy) and obtaining a clean 29/29 E2E run.