The Cache That Became a Database

Somewhere in the last decade the default answer to "we need a cache" stopped being memcached and started being Redis, and I cannot point to the argument where that was decided. There was no migration document. It happened the way defaults change: a new service picked Redis because the last new service had, the older memcached fleet stopped growing, and eventually the diagram had one box in it that used to have a different name.

But the two are not the same kind of component. Memcached is a cache and nothing else, and it behaves like one: volatile, small, honest about being lossy. Redis is a data structure server that is very good at caching, which is a different sentence. Lists, sets, sorted sets, hashes and streams are not cache features. They are reasons to keep the only copy of something in there.

So the features got used, because they were good and they were right there. Work queues, session state, rate limiter counters, leaderboards, distributed locks, feature flags, the in-flight state of a multi-step checkout. Persistence and replication existed, which made all of it feel safe. And here is the whole piece in one sentence: a great deal of production now depends on a component everyone classified as a cache, and therefore nobody built a recovery path for.

Memcached refuses almost everything. The data model is a key, a flat value and an expiry. The commands are roughly get, set, add, delete and increment. There are no types, no queries, no scripting, no ordering, no way to ask a question more interesting than "do you have this". You cannot build a queue in it because there is nothing to build a queue out of. The narrowness is not an omission. It is the product.

Redis is a server for data structures. Lists with blocking pops, sets with intersections, sorted sets with ranked ranges, hashes, bitmaps, streams with consumer groups, transactions, Lua, keyspace notifications, pub/sub. Every one of those is a small invitation to store something whose only copy is now in memory. Nobody sets out to keep the authoritative record in a cache. You set out to use a sorted set, and the sorted set turns out to be the authoritative record.

Ask what happens if the instance comes back empty. Not degraded, not slow. Empty, as though it had just been installed, because that is what a restart without persistence produces and what a failover to a cold replica can produce too. Go through the key prefixes one at a time and answer out loud.

For some of them the answer is that traffic falls through to the database and the database has a bad few minutes. That is a cache, and it has an availability requirement rather than a durability one. The failure is a thundering herd, and the fixes are known and cheap: coalesce the requests that miss on the same key, jitter the expiry times so a whole generation of entries does not expire in the same second, serve stale while you revalidate. Unpleasant, survivable, testable.

For others the answer is that the data is gone and there is nowhere to get it back from. The half-finished checkouts are gone. The queued jobs are gone, and nothing anywhere recorded that they were supposed to happen. The rate limiter counters are gone, which means every client gets a fresh allowance at the same instant. That was never a cache. It is a small database that has been running without a backup since the day it was installed.

The reason this matters more than it sounds is that both kinds of key usually live in the same instance, under the same memory limit and the same eviction policy. An allkeys-lru policy will happily evict a pending job to make room for a rendered page fragment, and it will not log an error, because from the server's point of view it did exactly what you configured. A cache miss is a slower answer. A queue miss is a job that never happened.

The append-only log is off unless you turn it on. The shipped redis.conf carries appendonly no. What you get by default is snapshotting: the documented save points fire after 3600 seconds if at least one key changed, after 300 seconds if at least a hundred did, and after 60 seconds if at least ten thousand did. That is a real durability mechanism and it is not a log. It is a photograph taken at intervals you did not choose.

Every second is not every write. With the log enabled there are three fsync policies: on every write, once a second, or never, and the default is once a second. The Redis persistence documentation is admirably direct about the consequence, which is that you may lose one second of data. One second sounds like nothing until you convert it: at the rate a rate limiter or a session store takes writes, one second is a large number of decisions that the recovered instance has no memory of.

Eviction happens before durability ever gets a turn. None of the above protects a key that was evicted while the server was healthy. If a memory limit is set with a policy that permits eviction, keys are deleted as a matter of routine operation, and the persistence layer faithfully records the deletion. Configure the limit with the default noeviction instead and writes start failing once memory fills, which is a different bad day. Both are correct behavior. Neither is what someone storing a queue there had in mind.

The idiom is four tokens long. SET a key with NX so it only takes if absent, with an expiry so a dead holder cannot wedge the system forever, and a random value so only the holder can release it. It works, it is fast, and it is one line, which is precisely why it ends up guarding things it should not.

In February 2016 Martin Kleppmann published an argument that the multi-node version was not safe for anything where correctness depends on the lock. His split is the useful part: a lock taken for efficiency saves duplicate work, and losing it costs you the duplicate; a lock taken for correctness prevents two writers from touching the same data, and losing it costs you the data. He argued that no lock service can supply that second guarantee on its own without a monotonically increasing token that the storage layer itself checks and rejects when stale.

Salvatore Sanfilippo answered in the same month and did not concede. His position was that the algorithm already hands out a large random token usable with a check-and-set, that it re-checks the elapsed time after acquiring a majority so unbounded message delay does not break it, and that a client which pauses after acquisition is a problem every auto-expiring lock has, not a flaw unique to this one.

I do not think that argument resolved, and I am suspicious of anyone who tells you it did. What I take from it is the question underneath, which nobody asks at the point where the lock is added: what does the resource do if two holders show up at once. If the answer is that the second one wastes some money, a single-node lock is fine and the debate is academic. If the answer is that the resource corrupts, then the resource has to reject the stale writer itself, and there is nothing you can configure in the lock server that will supply that for you.

It uses every core you paid for. A multithreaded server on a machine with a lot of cores absorbs a lot of load in one process. Getting the same throughput from a single-threaded engine means running more instances, which means sharding, which means a client that knows the topology, a cluster mode or a proxy, and a rebalancing procedure to rehearse. That is a real operational surface, and it appears the moment one node stops being enough. Memcached's answer to a bigger box is to use it.

It spends less memory on not being your data. Memcached's per-item overhead is small and the slab allocator gives it a bounded, predictable memory profile: the worst case is knowable because the size classes are fixed. Rich data structures cost more per entry, and at a hundred million small keys that difference stops being a rounding error and starts being the node size you have to buy.

The simplicity is a consequence of the refusal. There is no persistence configuration in memcached to get wrong, no scripting runtime to audit, no permissions model to maintain, no replication topology to reason about during a failover. That is not because its authors were less ambitious. It is the same property from the other side: the reason nobody ever accidentally built a database on memcached is that there was nothing in there to build one out of. Restraint in a dependency is a feature you only notice you were buying after you stop.

Redis 7.0 went GA on the twenty-seventh of April, six weeks before I am writing this. The headline items in the release notes are Functions, a new way to extend the server with scripts that live in the server rather than being shipped by each client; an access control overhaul with fine-grained key-based permissions and selectors so one user can hold several sets of command rules; sharded pub/sub in cluster mode; and a rewritten append-only file mechanism.

Read that list without the product name attached. A server-side programming model, a permissions system with per-key granularity, and a redesigned durability mechanism. Those are the things a database ships in a major version. A cache ships eviction policies and protocol efficiency. This is not a criticism of the release, which is a good one. It is the clearest available signal about what the thing has become.

Which is the right thing for Redis to be doing. It is building the product people already use it as, and it has been honest about that for years. The gap is not in the software. It is between what the software is and what your runbook still calls it, and only one of those has been updated since the migration.

Inventory by key prefix, not by instance. The unit of classification is the key pattern, because one instance holds several categories at once. List the prefixes, and for each one write the answer to the empty question in a sentence. acme:frag: regenerates from the database. acme:jobs: does not regenerate from anything. Two lines of documentation, and the second one is now a fact somebody can act on instead of an assumption sitting in one engineer's head.

If it is a queue, decide that on purpose. A list with a blocking pop is a queue the way a cardboard box is a filing cabinet. What you lose is acknowledgment, redelivery on consumer death, dead-letter handling and visibility into depth and age. Redis streams with consumer groups supply most of that and are a legitimate answer, as is a real broker. The failure mode is not picking the wrong one. It is arriving at one by accident and finding out during an incident which properties you never had.

I want to be careful not to turn this into nostalgia. The migration was mostly right, most teams should not reverse it, and Redis earns its operational surface for anyone actually using what it offers. The mistake was never the choice of software. It was that the choice got made once, for caching, on the evidence available for caching, and was then extended silently to a dozen jobs nobody re-evaluated it for, because the box in the diagram never changed its name.

The habit I would keep is the ten-second one. Every stateful component in your system has an answer to what happens when it is empty, and the answer is not a matter of opinion, and almost nobody has said it out loud. The empty instance is going to ask eventually, on a schedule you do not control, and it will be very interested in whether your backup policy agrees with the label on the box.