Thots on memcache redundancy

[Jed Reynolds]

Two thoughts on memcache redundancy.

If you have an active-passive failover with heartbeat, you can write a 
very simple memcache script on the passive node to poll the active 
node's memcache and keep a hot copy of it. This is the simplest 
replication you can do.

Another way, if you want to manage it at the application level, is to 
have a side channel to administrate availability of memcache 
repositories by managing access to memcache keys through the memcache 
client. You would do this by marking a memcache server as "deactivating" 
and each time an entry in the deactivating cache is requested, it either 
gets moved to another memcache server or regenerated. When you have an 
allowable threshold of "transitioned" keys in your deactivating memcache 
server (such as would not spike your database), you just purge the rest 
of the keys by removing the reference to the deactivated server. I think 
some memcache clients hash your keys across an array of servers, so 
removing a server means you have to adapt that behavior in the memcache 
client.

where the old server location would likely be
$serverindex = intval( $key ) % count( $oldservers_list );

and if that key exists, pull it and rehash it to a new hash entry:
$serverindex = intval( $key ) % count( $newservers_list );

You could keep a count of how many keys have been mapped to a server 
entry, and during the transition, you could create a secondary count of 
moves. Since you probably don't want to manage a list of keys and 
duplicate the logic of when they expire,  probably you'd just have to 
guess what ratio to kill it off at.

Or, your memcache client would have to go thru and rehash the mapping 
from the deactivating server to the remaining servers, moving every 
entry. This might have to be done such that you don't over-fill adjacent 
machines and force out too much of their entries, and don't incur undo 
load doing a lot of really fast transfers.

Jed