Skip to content

Redis blob cache assumes allkeys-lru; under noeviction a memory-full outage is permanent, not transient #188

Description

@Hazzng

Follow-up to #167 / PR #185 (recommendation 4 there, left out to hold scope).

Summary

Nothing in the code or the docs requires the Redis instance to run maxmemory-policy allkeys-lru. With the default noeviction, a Redis that reaches maxmemory starts refusing writes and never recovers on its own, because blob cache entries carry a 24 h TTL (REDIS_BLOB_CACHE_TTL_MS, default 86400000).

The harness measured this as the worst of the three #167 injections:

redis-cli -p 6380 CONFIG SET maxmemory 30mb   # + noeviction
-> 97.6% 5xx, never recovers

versus a 6 s CLIENT PAUSE, which recovers as soon as the pause lifts.

Why it matters

PR #185 split the Redis connection by role and scoped the circuit breaker, which bounds the blast radius of a stalled Redis. It does not help a full one: with noeviction the data plane stays refused until a human flushes keys or raises maxmemory, and the 24 h TTL means waiting it out is not a strategy.

This is a deployment-configuration requirement that currently exists only as an unstated assumption.

Suggested fix

  • Document maxmemory-policy allkeys-lru as a requirement for the Redis backing the blob cache, in CLAUDE.md and README.md alongside REDIS_URL / REDIS_DATA_URL.
  • Consider asserting it at boot: CONFIG GET maxmemory-policy and log a loud warning (not a hard failure — managed Redis providers sometimes disallow CONFIG GET).

Not verified

The maxmem variant was not re-run after PR #185. The 97.6% figure predates the role split, so the post-split number is unknown — the recovery behaviour, however, is a property of the eviction policy and the TTL, not of the connection topology.

Provenance

Surfaced by the #167 implementation while verifying the role split against the harness.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions