Practical limit on number of memcached servers in a single pool?

[Steven Grimm]

We have about 400 memcached instances split across two data centers with 
some, but not a ton of, cross-data-center traffic. Each memcached serves 
a couple tens of thousands of clients (Apache processes) -- memcached 
itself handles that number of connections just fine, though there are 
other side effects such as lots of memory going to TCP buffers that 
caused us to implement the UDP support.

There don't appear to be any particular scaling bottlenecks in terms of 
adding more memcached hosts. The only downside is that the more hosts 
you have in a pool, the less batching of "get" requests (multiget) is 
possible, and multiget turns out to give you a pretty substantial gain 
in CPU efficiency. My benchmarks showed that fetching 8 keys in a 
request only eats about 15% more CPU time than requesting one key.

One thing we do to address that problem is to identify types of objects 
that we tend to request in large batches. Each such object type is 
restricted to a particular slice of our memcached pool, increasing the 
amount of batching we do. Objects that tend to get requested one at a 
time or in small batches get spread out across the whole pool. Obviously 
we have to balance batching efficiency with overall request volume, 
susceptibility to a memcached host dying, etc., but that's the general idea.

-Steve


EKC wrote:
> What is the practical limit on the number of servers I can have in a
> memcached pool? 10's, 100's, 1000's? I'm using a gigabit switched LAN
> with all of the servers on the same subnet. I say thousands because I
> am using many virtual vmware servers.
>
> What is the largest number of servers that someone has used in a 
> memcached pool?
>
> My strategy for dealing with crashed servers is to treat them as
> cache-misses (no re-hasing of keys to servers). I'm using the Perl
> API.
>
> Also, if I build a new server and want to add it to an already running
> pool, can I do that without restarting memcached on each of the
> servers in the pool? Can I do a rolling restart? So, if I want to add
> server n+1 to a pool of n running memcached servers, I would do:
> 1) start memcached on server n+1 with a pool list of all n+1 servers
> 2) for each server 1...n, add server n+1 to the pool list and restart 
> memcached
>
> Or, is there better way of adding a new server to a running pool?
>
> I'm not too concerned with cache misses while memcached is restarting.
>
> Thanks in advance!