Why not let the fs handle this ??

[Steven Grimm]

Brian Moon wrote:
> For us, if just one server goes down, the other servers still server
> their normal stuff.  Only the keys that would have gone to/from the down
> server are reshuffled.

We did that at one point, but it has a big data consistency problem. Say 
host A normally holds a key. It goes down, so you put the data on host 
B. Now host A comes back up and you update the data; since host A is up, 
the key points to it and you stick the data there. Now host A goes down 
again and a client asks for the key. It will get the stale data from host B.

In practice this didn't bite us all that often, but it did happen (due 
to flaky hardware on one of our memcached servers). We ended up with no 
reshuffling of keys at all.

-Steve