Question concerning cpu utilization and memcached...

[Russ Garrett]

Christopher Gillett wrote:

> I configured memcached to run on 9 servers at 250 mb/server.  When the 
> cache is loaded, it looks like all instances are fairly evenly loaded 
> (memory stats are virtually identical for all instances).  However, 
> when we start running get()s I note that 1 of the 9 machines gets 
> extremely busy in terms of CPU utilization, while the rest of the 
> machines load fairly evenly.  For example, in one stress test I ran, 
> the CPU utilization across 8 nodes averaged 5-6%.  One node, however, 
> had utilization approaching 80%.  I am worried about having CPU 
> utilization issues deploying this into production as we are running 
> many CPU intensive processes on the same set of machines.  I also 
> noticed this same behavior early on testing with just two instances of 
> memcache running.
>
We *might* be seeing this as well. Sometimes one of our memcacheds 
decides to start randomly refusing some connections and apparently it's 
also failing to do get()s (although the PHP client we're using doesn't 
tell me what the error is, I need to hack it so it does). When I check 
on the memcached process it's using 30%+ CPU usage as opposed to ~3% on 
all the others, and a restart fixes everything. This is on Linux.

It's more of a hunch than anything though. It's not yet become annoying 
enough for me to look at it in any depth.

Russ Garrett
russ at last.fm