Transparent failover and restore?

[Greg Whalin]

Brad Fitzpatrick wrote:
> On Sat, 18 Dec 2004, Jon Valvatne wrote:
> 
> 
>>I think the point is this: If you're using memcached in a way that means
>>losing a server will bring down the system, then you should probably be
>>looking at using the MySQL "redundant storage thing" instead of the
>>memcached "cache thing". Traditional usage of a cache usually means
>>losing parts or all of it is *not* fatal.
> 
> 
> Exactly.
> 
> Or do what LiveJournal does:  have a dozen memcached machines, each with
> about the same amount of cache.  So if you lose a machine, you only lose
> 1/12th of the cache, until it re-hashes and populates over the next 10
> minutes.
> 
> I suppose I see the arguments from the people who only have 2 machines
> and want good cache hit rate performance in the face of failure, but the
> cache coherency problems just seem too hard to re-implement when solutions
> already exist.
> 
> Perhaps memcached atop NDB cluster (what MySQL Cluster is built on)
> would be interesting.  NDB would do the cache coherency across the
> cluster, and memcached could simply do the protocol handling and LRU
> maintenance and cache discarding...
> 
> - Brad

I for one would not want this complexity moved into the server.  I like 
memcached for what it is.  That being a high speed distributed object 
cache.  The more complexity added in at the server side the slower it 
will be.  Given there are several ways to accomplish this without adding 
the complexity to the server, seems like a waste to move into server 
side.  And, given that the hardware for these machines need not be 
expensive, you could build a large cluster of servers without much $$$ 
invested.

I also feel this discussion is purely academic as the only time our 
memcache servers go down is when we shut them down.  The server is 
extremely reliable in our experience.  Adding in more server side 
complexity could also potentially hurt this reliability as well.