Single Redis connection couples blob cache to locks, causing replica-wide 503 storms

[Hazzng]

Summary

One ioredis connection carries everything: the blob cache, the path snapshot, the version counter, and both distributed locks (src/redis/client.ts:32 is a singleton; src/api/server.ts:57 and :153 hand that same client to RedisPathSnapshot and RedisBlobCache). Large blob writes queue head-of-line in front of latency-critical lock and version commands, and a process-wide circuit breaker then fast-fails every request on the replica.

Measured coupling (direct proof)

260 concurrent 2 MiB SETs in flight:

	Shared connection	Separate connection
INCR	fails, Command timed out at 2043 ms	OK in 44 ms

At 60 queued blobs: shared INCR 1240 ms vs isolated 12 ms.

The resulting outage

A 6-second Redis stall is enough to produce a replica-wide 503 storm:

Fault injection	5xx	Window	p50
CLIENT PAUSE 6000	7,572 / 11,000 (66.8%)	7.1 s	7 ms
maxmemory 30mb + noeviction	54,312 (97.6%)	never recovered	7 ms
docker pause 70 s	114,213 (98.8%)	70.9 s	6 ms

Chain: blob set times out (commandTimeout: 2000, src/redis/client.ts:38) → the thrown error calls breaker.recordFailure() in the lock acquire paths (distributed-rw-lock.ts:198,292,326) → the process-wide breaker (src/redis/circuit-breaker.ts:160-167, threshold 5) opens → if (breaker.isOpen()) throw new LockAcquireTimeoutError(...) fires before any Redis I/O → ELOCKTIMEOUT → 503 in 1–7 ms.

Notes:

• Reads do not survive it. Bystander GET /files and readOnly:true execs 503 at the same rate — the shared acquire checks the same breaker.
• Blast radius is the whole replica, every tenant, because the breaker is one singleton.
• It self-heals sub-second once Redis answers (openMs = 5000, then a probe), but the duration is unbounded while the Redis condition persists — maxmemory + noeviction gave 100% 503 indefinitely, since blob TTL is 24 h and nothing frees.
• 114,213 503s produced zero log lines beyond request logs, which is why this was originally misdiagnosed.

Recommended fixes

1. Split the Redis connection by role (highest value, directly proven above): a second client for the data plane (blob cache, path snapshot), with locks/version/session staying on the control-plane client.
2. Backpressure the blob cache: RedisBlobCache.set is called fire-and-forget (void this.#blobCache.set(...) at dialects/postgres.ts:213,610,632,649,666,725) with no in-flight cap. Add a bounded in-flight semaphore (count and bytes) and drop cache writes over the cap — the cache is fail-open by contract. Consider lowering REDIS_BLOB_MAX_BYTES below 2 MiB.
3. Scope the circuit breaker per role so a data-plane problem can never fast-fail the lock path, and emit redis_circuit_open / redis_circuit_closed events so this is diagnosable.
4. Require an eviction policy on the blob-cache Redis (allkeys-lru) and document it — with noeviction the outage is permanent rather than transient.

Provenance

Found during pre-merge load testing of #162. Not introduced by that PR.