Why not let the fs handle this ??

[Brian Moon]

Don MacAskill wrote:
> 1. When your server comes back up, you don't have to worry about stale 
> data being left on old servers should it crash again.

We worried about this at first, but then, after using memcached for a 
while, we forgot about it.  The simple fact is, for us, the only time 
the servers are down is when we are working on them.  memcached does not 
die on its own.  Or has yet to do so AFAIK.  So, we can orchestrate a 
flush across the full pool if we are worried about stale data.  We have 
maybe done that once or twice.

Perhaps your application has data changing by the second or is mission 
critical data (e.g. medicalor financial).  We do not cache data that 
critical.  Anything that has to be correct and up to the second comes 
straight from the database.

Don MacAskill wrote:
> 2.  You're only out 1/100th of your cache, so the other 99 continue on 
> as before and don't get hammered while keys rebalance and the entire 
> cache is rebuilt.
> 
> The key is to keep that downed 100th machine in your pool, so the key 
> allocation algorithm still "counts" it, but to somehow let your 
> application know not to write to it while it's in a downed state.

The PHP/PECL client handles all that for us.  We just leave the down 
server in the list and it works around it.

Don MacAskill wrote:
> Strangely, we seem to be fairly unique in this approach.  At least, when 
> I mention it on the list, people don't seem to get it.  To my mind, it's 
> the only sane way to deal with the problem.

Only in the approach that you stale data causes a problem.  I think most 
people operate without having all their keys rehashed.  At least, that 
is how the PHP/PECL client handles things.  And from my understanding, 
the way the Perl client does as well.

Steven Grimm wrote:
 > In practice this didn't bite us all that often, but it did happen (due
 > to flaky hardware on one of our memcached servers). We ended up with
 > no reshuffling of keys at all.

If we had a server doing that, we would likely take out of production 
service until it could be repaired.  In fact, that is what we do. 
Servers going up and down are no fun.  I know not everyone has that 
luxury.  Just clearing up my POV.

We have been using memcached for nearly a year now and I can honestly 
say that I have never gotten one report of stale data.  Again, our 
cached content is largely static once created (much like LJ I would 
guess) and not mission critical (eg. medical or financial).

-- 

Brian Moon
-------------
http://dealnews.com/
Its good to be cheap =)