Memcached and maximum persistent connections
sgrimm at facebook.com
Sun Oct 7 02:38:01 UTC 2007
64-per isn't *that* many if you are using multi-CPU web servers and
they're spending most of their time waiting for the database or
memcached. We've run with higher limits than that in the past. Right now
I think our limit is around 50 Apache processes per host.
We used to try to figure out our total maximum number of incoming
connections and specify the corresponding value on memcached's command
line, but after one too many instances of, "We just added another 300
web servers, forgot to bump up the limit, and now we're getting
connection failures," we decided to run memcached with "-c 500000" so as
to avoid running into any connection limits in the foreseeable future.
It stinks to have to restart memcached just to bring a bunch of new web
hosts online. Running with that high limit has not caused any problems.
The growth beyond the memory limit can be more than just a wee bit, BTW;
for servers with lots of non-UDP clients, we set aside a couple
gigabytes for per-connection memory. It's not just the
memcached-initiated memory allocation you have to worry about, but also
the kernel-managed TCP buffers. And even inside memcached, the amount of
memory used per connection can be variable; memcached allocates enough
memory to buffer up an entire incoming command, so if you do a huge
multiget it will (temporarily) eat a lot of memory internally.
> 64-per is a lot :)
> Anyway, it's fine, go nuts. There's a comment next to the maxfiles
> /* to limit connections-related memory to about 5MB */
> ... I have no idea how they arrived at this conclusion but it's not
> hard to figure out. The only things to keep in mind:
> - There are small memory allocations per connection.
> - They're not presently tracked by the global memory limit, so
> memcached will grow beyond that limit a wee bit based on your
> connections (and I'm not man enough to fix it instead of just
> repeating this).
> Robin Schuil wrote:
>> Hi all,
>> I'm trying to run memcached but had some trouble with the connection
>> limit. Already found that I need to use the '-c <num>' option to
>> increase the # of maximum simultaneous connections. However, I have
>> 200 webservers each with ~64 apache childs running (there may be
>> peaks). If each process would create a persistent connection that
>> would mean that each of my 10 memcached nodes have to handle 12,800
>> simultaneous connections.
>> I guess that shouldn't be a problem, but the low default maximum of
>> 1024 somewhat bothers me. Is it sane to start memcached with a limit
>> of 40K connections, or is it likely that I'll run into other problems
>> Your input is greatly appreciated.
More information about the memcached