Active Health Checks

[moveson]

Problem

When app2 was added to the load balancer today, it failed to boot due to the
database firewall not having its IP whitelisted. Puma was running and holding
port 9000 but not accepting connections, causing Caddy on app2 to return 502
for every request.

Caddy's passive health checks failed to remove app2 from rotation because:

1. The 502 responses were being served by Caddy on app2 itself, not the
   upstream, so the load balancer saw them as successful responses
2. The passive check required 10 failures within 10 seconds before marking
   a backend unhealthy

This caused significant downtime for opensplittime.org.

Solution

1. Add a /up health check route to Rails using the built-in health controller:

   In config/routes.rb:

 get "up" => "rails/health#show", as: :rails_health_check

This checks the database connection and returns 503 if it is unavailable.

2. Add active health checks to the Caddy load balancer config on elbee so it
   probes each backend every 10 seconds independently of real user traffic:

 "health_checks": {
 "active": {
 "path": "/up",
 "interval": 10000000000,
 "timeout": 5000000000
 },
 "passive": {
 "fail_duration": 10000000000,
 "max_fails": 3
 }
 }

Note: Caddy config on elbee is managed via its API, so this would need to
be applied via the Hatchbox "Update Caddy" mechanism or persisted through
Hatchbox's config system.

Checklist

• Add /up route to config/routes.rb
• Verify /up returns 200 in production
• Configure active health checks in Caddy on elbee
• Add DigitalOcean database firewall IP whitelisting to the new server
  checklist

[moveson]

Health check design decision (PR #2156, merged)

The /up route was added using the stock Rails::HealthController (get "up" => "rails/health#show"), deliberately without a database check. Summary of the trade-offs considered:

Why not a DB-querying health check?

A version that ran SELECT 1 per probe was considered and rejected. Note that, contrary to the issue text above, the stock controller does not check the database — it returns 200 once the app has booted and only 503s if rendering raises.

• Both versions catch the app2 incident as it happened. Puma held port 9000 but the app never finished booting, so an active probe fails on connection hang/reset regardless of controller logic.
• The DB check's risk is correlated failure through the shared database. All backends share one Postgres, so a DB blip — or merely a slow DB under load — could time out every backend's probe at once, causing Caddy to mark all upstreams unhealthy and 503 everything. Probes timing out under load can also pull healthy-enough backends from rotation, concentrating traffic on the rest and cascading (probes compete for the same connection pool as real requests).
• The stock check's residual risk is a false-healthy: a server that boots successfully but can't reach the DB (e.g., a future firewall gap where boot doesn't happen to hang) would answer 200 while 500ing real traffic. The durable fix for that class is the new-server provisioning checklist item, with active health checks as defense in depth.
• If the false-healthy scenario ever becomes a concern, the discussed alternatives were: (a) a production initializer that verifies the DB connection at boot so "booted but DB-less" can't exist, or (b) a DB check with a cached/damped result (~30–60s) to avoid probe-driven cascades.

Conclusion: the stock check can never take the whole fleet out of rotation, requires no custom code, and is sufficient for the boot-failure class of incident.