how to tell when your cache is full?

[Ben Hartshorne]

Hi,

I am trying to evaluate the schedule for expanding our cache.  We have
several different types of data all getting thrown into one large
central cache (made up of 1 instance on each of 4 machines).  Some of it
is transient (caching 'popular' data for 5 minutes) and some of it is
more permanent (1wk expiration time).  All data is backed by permanent
storage for cache misses.  On cache miss, the data is repopulated into
memcache from permanent storage.  

My problem - the 'stats' command has a metric 'curr_items' that reports
the current number of items stored.  However, when a piece of data
expires, that counter is not decremented until you issue a 'get' on the
data and fail.  

In order to cache 'popular' data, all data of that type is cached,
assuming that the popular ones will be hit and updated within the
expiration time, and so remain in the cache, while the unpopular data
will just expire and nobody cares.  

The problem with the curr_items stat is that if I ever get a cache miss
on the transient data, I immediately fetch it from the database and
stick it back in the cache, causing the curr_items to decrement and then
increment again.  Data that is unpopular is stored (causing an
increment) but never retrieved, so the curr_items never decrements.  The
effect is a monotomically increasing number in curr_items until it tops
out (at 2352883, though I'm not sure what's special about that number).  

Because of the different types of data and the changing popularity of
data, cache hit percentage is not a good proxy for telling me when my
cache has filled up.

At the moment, I am pretty sure the cache is not full because the more
persistent data (1wk expiry time) usually sticks around, though I don't
have a good metric to prove that, it seems to be the case.

How do I tell when my cache is full and I need to add the 5th server?  I
tried watching the memory utilization as reported by the OS but it is
also monotomically increasing until it tops out at the limit given to
memcache (7GB, in this case).


Thanks for any advice you might have,

-ben


-- 
Ben Hartshorne
email: ben at hartshorne.net
http://ben.hartshorne.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://lists.danga.com/pipermail/memcached/attachments/20070524/02033fa7/attachment.pgp