Practical limit on number of memcached servers in a single pool?
sgrimm at facebook.com
Tue Oct 24 22:04:29 UTC 2006
We have about 400 memcached instances split across two data centers with
some, but not a ton of, cross-data-center traffic. Each memcached serves
a couple tens of thousands of clients (Apache processes) -- memcached
itself handles that number of connections just fine, though there are
other side effects such as lots of memory going to TCP buffers that
caused us to implement the UDP support.
There don't appear to be any particular scaling bottlenecks in terms of
adding more memcached hosts. The only downside is that the more hosts
you have in a pool, the less batching of "get" requests (multiget) is
possible, and multiget turns out to give you a pretty substantial gain
in CPU efficiency. My benchmarks showed that fetching 8 keys in a
request only eats about 15% more CPU time than requesting one key.
One thing we do to address that problem is to identify types of objects
that we tend to request in large batches. Each such object type is
restricted to a particular slice of our memcached pool, increasing the
amount of batching we do. Objects that tend to get requested one at a
time or in small batches get spread out across the whole pool. Obviously
we have to balance batching efficiency with overall request volume,
susceptibility to a memcached host dying, etc., but that's the general idea.
> What is the practical limit on the number of servers I can have in a
> memcached pool? 10's, 100's, 1000's? I'm using a gigabit switched LAN
> with all of the servers on the same subnet. I say thousands because I
> am using many virtual vmware servers.
> What is the largest number of servers that someone has used in a
> memcached pool?
> My strategy for dealing with crashed servers is to treat them as
> cache-misses (no re-hasing of keys to servers). I'm using the Perl
> Also, if I build a new server and want to add it to an already running
> pool, can I do that without restarting memcached on each of the
> servers in the pool? Can I do a rolling restart? So, if I want to add
> server n+1 to a pool of n running memcached servers, I would do:
> 1) start memcached on server n+1 with a pool list of all n+1 servers
> 2) for each server 1...n, add server n+1 to the pool list and restart
> Or, is there better way of adding a new server to a running pool?
> I'm not too concerned with cache misses while memcached is restarting.
> Thanks in advance!
More information about the memcached