Production platform: deployment architecture, nightly eval canary, capacity review

[BigSimmo]

Summary

Three production-platform workstreams, per the branch brief. No src/lib retrieval/answer code touched (parallel chats own those files); no live Supabase settings touched.

1. Deployment topology (docs/deployment-architecture.md + Docker images)

• App tier decision: single warm Next.js container (Node 24) on a container host in Sydney, co-located with Supabase sjrfecxgysukkwxsowpy (ap-southeast-2, confirmed via read-only project lookup). Fly.io syd recommended; serverless explicitly rejected (in-memory answer coalescing/caches are load-bearing, fire-and-forget post-response work, 30 s strong-route requests, auth 10-conn amplification).
• Ingestion decision: containerize the existing worker (Dockerfile.worker) rather than completing the edge-agent migration — PyMuPDF/Tesseract are native binaries that cannot run in Deno isolates, jobs run multi-minute against edge ceilings, and claim_ingestion_jobs is already multi-instance-safe. indexing-v3-agent stays as the completion/repair gate.
• Queue durability: documented from claim_ingestion_jobs semantics — no heartbeat; locked_at set once at claim; 45-min stale reclaim; attempts consumed at claim, so crash-looping jobs exhaust max_attempts into terminal failed (implicit dead letter, recovered via recover:ingestion under the rollback fence). Rule: stale window must exceed p100 job runtime; heartbeat-on-progress-update flagged as backlog.
• Secrets matrix + staging plan (dedicated second Supabase project, not a branch; check:supabase-project will need env-aware expected-project config when staging is provisioned — deliberate speed bump, documented).

Verified:

• docker build . → npm run build compiled successfully inside the container (gate) ✅
• Image boots and serves: /api/health 200 (status: ok), / 200, ready in 563 ms ✅
• Worker image: tesseract 5.3.0, import fitz, pytesseract, PIL OK, tsx on Node 24.18.0 ✅

2. Observability + eval canary (docs/observability-slos.md, .github/workflows/eval-canary.yml)

• SLOs with warn/page thresholds: answer p95 by routing_mode, source-gap rate, unsupported rate, hybrid_rpc_errors rate (near-zero tolerance — this is the silent-degradation guard), degraded/source-only rate. Measurement SQL over rag_queries.metadata included.
• Nightly canary (18:00 UTC = 02:00 AWST) + workflow_dispatch: secret preflight → check:supabase-project → eval:retrieval:quality --fail-on-threshold → eval:quality --rag-only --limit 8 --fail-on-threshold. On scheduled failure it opens/updates a GitHub issue labeled eval-canary. This makes the golden eval (which PR CI can never run) a standing nightly guard.
• Verified locally: required secrets exist in the repo (SUPABASE_SERVICE_ROLE_KEY, OPENAI_API_KEY, E2E_USER_EMAIL), and check:supabase-project passes with exactly the env the workflow sets.
• ⚠️workflow_dispatch cannot be tested pre-merge — GitHub returns 404 for workflows absent from the default branch (attempted: gh workflow run eval-canary.yml --ref claude/production-platform). First post-merge step: gh workflow run eval-canary.yml and confirm green before trusting the schedule. This is called out in the workflow header and the SLO doc.

3. Capacity review (docs/capacity-review.md, scripts/soak-test.ts)

• 30-clinician ward-round model vs the auth 10-connection cap, PostgREST/Supavisor pooling, ~6-RPC per-answer fan-out, OpenAI limits, and existing coalescing.
• First hard failure: the auth 10-connection cap under sign-in/refresh bursts (fix: percentage-based allocation in the dashboard — operator action, needs explicit approval). First soft failure: Postgres CPU under hybrid-RPC concurrency (p95 inflation before errors — hence the latency SLOs).
• scripts/soak-test.ts: staging-only ward-round load driver. Guards verified: refuses without --target, refuses without --confirm-staging, refuses hosts matching the production project ref. Full loop exercised end-to-end against a local container (fixture queries loaded, 75/25 search/answer mix, percentile summary, 5% failure gate). Not run against production.

Gates

• verify:cheap ✅ (lint, typecheck, sitemap, 1143/1143 unit tests)
• App Docker image build (contains npm run build) ✅; container boot smoke ✅
• Worker Docker image build + OCR stack smoke ✅
• format:check: new files clean; the only warning is pre-existing local CRLF noise on CLAUDE.md (byte-identical to main)

Post-merge checklist

1. gh workflow run eval-canary.yml → confirm green before trusting the nightly schedule.
2. Operator decisions when ready: host account (Fly.io syd), staging Supabase project, auth percentage-based pool allocation (ask-first — live settings).

🤖 Generated with Claude Code

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[BigSimmo]

⚠️Heads-up before the post-merge canary dispatch test: live Supabase keys were rotated/disabled on 2026-07-06 (a concurrent session's finding — legacy service-role disabled, the Jul-5 sb_secret unregistered). The repo secret SUPABASE_SERVICE_ROLE_KEY was last set 2026-06-30, so it is likely stale now.

Concretely:

1. Update the SUPABASE_SERVICE_ROLE_KEY repo secret to the current valid secret key before running gh workflow run eval-canary.yml, or the first dispatch will fail at the eval admin-client step (and CI's main-branch deployment boot smoke will hit the same wall).
2. The canary's secret preflight only checks presence, not validity — a stale key fails at the Supabase health probe inside the eval (assertSafeToRunEvals), which is still a loud, attributable failure, but easy to misread as a retrieval regression. If the first dispatch fails there, check the key first.

[BigSimmo]

✅ Post-merge canary dispatch test done — run 28919595072.

Confirms the workflow wiring end-to-end. Every step up to the eval passed — secret preflight, Node setup, npm ci, and the Supabase project-identity guard. It then failed loud at the golden-retrieval step with Unregistered API key, and the issue-open step correctly skipped (that fires only on schedule, not workflow_dispatch).

This is the expected outcome given the stale secret: SUPABASE_SERVICE_ROLE_KEY is still dated 2026-06-30 while live keys rotated 07-06. One action to a green nightly canary: update that repo secret to the current valid key, then re-dispatch (gh workflow run eval-canary.yml) to confirm green before the schedule is trusted.

[BigSimmo]

🔑 Secrets update confirmed working. Re-dispatched the canary after the 07-08 secret refresh — run 28939769131.

The Unregistered API key error is gone. The eval now runs against the live corpus and retrieval cases pass (clozapine-anc-threshold, monitoring-threshold-from-chart, alcohol-ciwa-threshold, clozapine-cbc-abbreviation all PASS with docRecall@5=1.00). So SUPABASE_SERVICE_ROLE_KEY + OPENAI_API_KEY + E2E_USER_EMAIL are all valid and correctly wired.

The run still exits 1, but now on a different, non-secret cause: OpenAI quota is exhausted. Falling back to a source-only answer. That's an OpenAI account billing/quota top-up, not a repo-config problem. Once quota is restored, re-dispatch (gh workflow run eval-canary.yml) should go green — the wiring is fully proven.