Memcached and maximum persistent connections

[Steven Grimm]

64-per isn't *that* many if you are using multi-CPU web servers and 
they're spending most of their time waiting for the database or 
memcached. We've run with higher limits than that in the past. Right now 
I think our limit is around 50 Apache processes per host.

We used to try to figure out our total maximum number of incoming 
connections and specify the corresponding value on memcached's command 
line, but after one too many instances of, "We just added another 300 
web servers, forgot to bump up the limit, and now we're getting 
connection failures," we decided to run memcached with "-c 500000" so as 
to avoid running into any connection limits in the foreseeable future. 
It stinks to have to restart memcached just to bring a bunch of new web 
hosts online. Running with that high limit has not caused any problems.

The growth beyond the memory limit can be more than just a wee bit, BTW; 
for servers with lots of non-UDP clients, we set aside a couple 
gigabytes for per-connection memory. It's not just the 
memcached-initiated memory allocation you have to worry about, but also 
the kernel-managed TCP buffers. And even inside memcached, the amount of 
memory used per connection can be variable; memcached allocates enough 
memory to buffer up an entire incoming command, so if you do a huge 
multiget it will (temporarily) eat a lot of memory internally.

-Steve


dormando wrote:
> 64-per is a lot :)
>
> Anyway, it's fine, go nuts. There's a comment next to the maxfiles 
> section:
> /* to limit connections-related memory to about 5MB */
>
> ... I have no idea how they arrived at this conclusion but it's not 
> hard to figure out. The only things to keep in mind:
>
> - There are small memory allocations per connection.
> - They're not presently tracked by the global memory limit, so 
> memcached will grow beyond that limit a wee bit based on your 
> connections (and I'm not man enough to fix it instead of just 
> repeating this).
>
> -Dormando
>
> Robin Schuil wrote:
>> Hi all,
>>
>> I'm trying to run memcached but had some trouble with the connection 
>> limit. Already found that I need to use the '-c <num>' option to 
>> increase the # of maximum simultaneous connections. However, I have 
>> 200 webservers each with ~64 apache childs running (there may be 
>> peaks). If each process would create a persistent connection that 
>> would mean that each of my 10 memcached nodes have to handle 12,800 
>> simultaneous connections.
>> I guess that shouldn't be a problem, but the low default maximum of 
>> 1024 somewhat bothers me. Is it sane to start memcached with a limit 
>> of 40K connections, or is it likely that I'll run into other problems 
>> then?
>> Your input is greatly appreciated.
>>
>> Thanks,
>>
>> Robin
>