Distribution of data based on traffic

[Steven Grimm]

We've seen that behavior, though not nearly on that scale (but then, we 
have literally hundreds of millions of objects in our cache, so that 
tends to even out any inequalities.)

Before going to the trouble of building a system that keeps track of 
which individual keys need to get redirected where -- and bear in mind 
that since server selection is a client-side operation, you have to keep 
that information in sync among all your clients -- I'd try just tweaking 
the hash function on the client side. Maybe shift off the low bit of the 
hash value or something like that. Unless the traffic difference is due 
to a single key getting hammered, that's probably all you need to do to 
spread things around.

-Steve


Darryl Kuhn wrote:
> We've been running memcached in our production environment for several 
> weeks now quite successfully but have noticed that volume of internal 
> traffic on our servers is quite unevenly distributed, varying anywhere 
> from 2-3Mbps up to 30Mbps. After looking into it a bit it was clear to 
> us that this was a result memcached gets. What we're finding is that 
> since a key is only stored on one server, keys with values that are 
> large, or more often the case values that are accessed frequently 
> cause one server to be accessed quite a bit more than others.
>  
> We can't put a key in more than one spot (nor do we want to), but the 
> more I've thought about it the more I believe that there might be some 
> value in a mechanism that self adjusts key locations by moving heavily 
> trafficked keys to less heavily trafficked servers.
>  
> Say for example that 10 keys are accessed every page request and the 
> hashing mechanism puts all 10 keys on a single server (say out of a 
> pool of 10 memcached servers). Over time (based on access rates or 
> some other suitable metric) this mechanism would ensure that those 10 
> keys are evenly distributed across all 10 memcached instances leveling 
> off network traffic.
>  
> It's a thought at any rate - has anyone else dealt with this kind of 
> issue?
>  
> Cheers,
>  
> Darryl Kuhn
> Chief Technology Officer
> Skinit.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/memcached/attachments/20061201/967bd6e3/attachment.html