Question concerning cpu utilization and memcached...

[Christopher Gillett]

Environment:  FreebBSD 4.9, memcached-1.1.12, libevent 1.0c,
python-memcached-1.2
Issue:  Very heavy CPU utilization during get() processing

I googled around and didn't find anything useful on this, so please be
gentle if this question is a repeat of a previous discussion.  :-)

I've spent the last couple days building out a memcached-based front end
for one of our systems.  It caches a little over 1 million items, where
the key is fairly short, and the content is also very short.  We query a
good sized database and load everything in cache, and then run all
lookups out of cache.  Performance has been excellent.  In testing in a
grid configuration with 30+ nodes I am getting 500-600 cache hits per
second on all nodes simultanesously which is really good (especially
considering that we're using Python to drive the process).  I do observe
one troubling behavior that I'm curious to know more about. 

I configured memcached to run on 9 servers at 250 mb/server.  When the
cache is loaded, it looks like all instances are fairly evenly loaded
(memory stats are virtually identical for all instances).  However, when
we start running get()s I note that 1 of the 9 machines gets extremely
busy in terms of CPU utilization, while the rest of the machines load
fairly evenly.  For example, in one stress test I ran, the CPU
utilization across 8 nodes averaged 5-6%.  One node, however, had
utilization approaching 80%.  I am worried about having CPU utilization
issues deploying this into production as we are running many CPU
intensive processes on the same set of machines.  I also noticed this
same behavior early on testing with just two instances of memcache
running.

Has this type of behavior been observed by others?  Any information as
to what causes this or how to avoid it?

Thanks for any insight....
Chris Gillett

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/memcached/attachments/20050428/151732ab/attachment.html