how to tell when your cache is full?

[Gavin M. Roy]

I would imagine when you start seeing more evictions in your stats than you
expect, you know your cache is full.  I'd say anything else is
over-complicating it.

On 5/25/07, Ben Hartshorne <memcache at green.hartshorne.net> wrote:
>
> OK, so the best I can come up with is to insert a new record every x
> minutes (say, 60), choose what threshold you expect or care about
> records not getting flushed due to overflowing the cache, wait that
> long, and then check the record and see if it's still there, verifying
> that your cache has not been blown.  This seems rather crude, and only
> tells you when you have already lost your cache, not when you are
> getting close to doing so (though you can double your time threshold and
> hope that's good enough).
>
> So for example, I am interested in making sure a record persists for 6
> hours.  I double that time to give a bit of a threshold so as to get
> some warning.  I insert 12 records, one every hour.  After all have been
> inserted, I check the 12th record before resetting it.  If it's still
> there, my cache has enough space that records can last 12 hours.  If
> it's gone, I can step through them to determine how much of the cache
> had to be freed to make room for the current data.
>
> Has anybody done this?  Maybe as a nagios plugin?
>
> This procedure does raise a question though, about the details of which
> records are dropped when the cache becomes full.  When the cache becomes
> full, how is it determined which records to drop?  Oldest access time?
> Oldest insert time?  Largest record?  I'm kinda hoping it's oldest
> access time, because that makes the most sense for our environment
> (since popular records will stay in the cache longer than unpopular
> records).  :)
>
> Thanks,
>
> -ben
>
>
> On Thu, May 24, 2007 at 12:49:54PM -0700, Ben Hartshorne wrote:
> > Hi,
> >
> > I am trying to evaluate the schedule for expanding our cache.  We have
> > several different types of data all getting thrown into one large
> > central cache (made up of 1 instance on each of 4 machines).  Some of it
> > is transient (caching 'popular' data for 5 minutes) and some of it is
> > more permanent (1wk expiration time).  All data is backed by permanent
> > storage for cache misses.  On cache miss, the data is repopulated into
> > memcache from permanent storage.
> >
> > My problem - the 'stats' command has a metric 'curr_items' that reports
> > the current number of items stored.  However, when a piece of data
> > expires, that counter is not decremented until you issue a 'get' on the
> > data and fail.
> >
> > In order to cache 'popular' data, all data of that type is cached,
> > assuming that the popular ones will be hit and updated within the
> > expiration time, and so remain in the cache, while the unpopular data
> > will just expire and nobody cares.
> >
> > The problem with the curr_items stat is that if I ever get a cache miss
> > on the transient data, I immediately fetch it from the database and
> > stick it back in the cache, causing the curr_items to decrement and then
> > increment again.  Data that is unpopular is stored (causing an
> > increment) but never retrieved, so the curr_items never decrements.  The
> > effect is a monotomically increasing number in curr_items until it tops
> > out (at 2352883, though I'm not sure what's special about that number).
> >
> > Because of the different types of data and the changing popularity of
> > data, cache hit percentage is not a good proxy for telling me when my
> > cache has filled up.
> >
> > At the moment, I am pretty sure the cache is not full because the more
> > persistent data (1wk expiry time) usually sticks around, though I don't
> > have a good metric to prove that, it seems to be the case.
> >
> > How do I tell when my cache is full and I need to add the 5th server?  I
> > tried watching the memory utilization as reported by the OS but it is
> > also monotomically increasing until it tops out at the limit given to
> > memcache (7GB, in this case).
> >
> >
> > Thanks for any advice you might have,
> >
> > -ben
> >
> >
> > --
> > Ben Hartshorne
> > email: ben at hartshorne.net
> > http://ben.hartshorne.net
>
>
>
> --
> Ben Hartshorne
> email: ben at hartshorne.net
> http://ben.hartshorne.net
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.1 (GNU/Linux)
>
> iD8DBQFGVySPKeT3tvTdv64RAt3iAKCioA54txWxS7HjNJeBvzEd50yzqwCcCFSl
> 7F6qx4H64dJkWQAzu9ptDlQ=
> =pAVS
> -----END PGP SIGNATURE-----
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/memcached/attachments/20070525/d42a7f82/attachment.htm