Distribution of data based on traffic
sgrimm at facebook.com
Sat Dec 2 04:16:00 UTC 2006
We've seen that behavior, though not nearly on that scale (but then, we
have literally hundreds of millions of objects in our cache, so that
tends to even out any inequalities.)
Before going to the trouble of building a system that keeps track of
which individual keys need to get redirected where -- and bear in mind
that since server selection is a client-side operation, you have to keep
that information in sync among all your clients -- I'd try just tweaking
the hash function on the client side. Maybe shift off the low bit of the
hash value or something like that. Unless the traffic difference is due
to a single key getting hammered, that's probably all you need to do to
spread things around.
Darryl Kuhn wrote:
> We've been running memcached in our production environment for several
> weeks now quite successfully but have noticed that volume of internal
> traffic on our servers is quite unevenly distributed, varying anywhere
> from 2-3Mbps up to 30Mbps. After looking into it a bit it was clear to
> us that this was a result memcached gets. What we're finding is that
> since a key is only stored on one server, keys with values that are
> large, or more often the case values that are accessed frequently
> cause one server to be accessed quite a bit more than others.
> We can't put a key in more than one spot (nor do we want to), but the
> more I've thought about it the more I believe that there might be some
> value in a mechanism that self adjusts key locations by moving heavily
> trafficked keys to less heavily trafficked servers.
> Say for example that 10 keys are accessed every page request and the
> hashing mechanism puts all 10 keys on a single server (say out of a
> pool of 10 memcached servers). Over time (based on access rates or
> some other suitable metric) this mechanism would ensure that those 10
> keys are evenly distributed across all 10 memcached instances leveling
> off network traffic.
> It's a thought at any rate - has anyone else dealt with this kind of
> Darryl Kuhn
> Chief Technology Officer
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the memcached