Transparent failover and restore?

[Gregory Block]

On 20 Dec 2004, at 21:08, Richard 'toast' Russo wrote:
> Memcached clients* will rehash the key if the server the key gets 
> hashed to is offline.  When you take one machine offline you'll lose 
> 1/5th of your cache, and when you bring it back, you'll again lose 
> 1/5th of your cache as the keys originally asigned to the that server 
> migrate back (when the clients notice the server is back).
>
> You could make the connection timeouts more agressive to reduce the 
> cost of the client finding dead servers.
>

...or implement what I originally discussed, in the client, which 
pushed updates to 2 servers instead of 1, where the second server is 
the server it would have chosen next if the first one were to be marked 
as "down"; when server 1 isn't down, servers 1 and 2 get updates; when 
1 goes down, 2 gets results; when 1 comes back up, query 1 and 2, and 
push the missing update into 1.

This *could* be done very, very painlessly, and it would eliminate the 
whole problem instantly.