Strange memcached behavior on stress test

[Garth Webb]

On Sun, 2006-06-11 at 18:45 -0400, Stephen Woodbridge wrote:
> dormando wrote:
> > 
> >> Debian 32bit Sarge with a HUGEMEM kernel patch and 6 GB of memory and 
> >> 1 GB configured for memcached. I grew to about 1.5 GB then started 
> >> throwing the errors.
> > 
> > What were all of the options you used to start memcached?
> 
> /usr/bin/memcached -m 1024 -p 11211 -u root -l 10.14.12.101
> 
> >> The stress test script generated a random sized data packet (1-20KB in 
> >> size) then does a set and an immediate get and compares the set and 
> >> and got value, it counts it as an error if they do not match. this was 
> >> done via a perl script. Sorry, I don't have more specific data, I 
> >> would have to rerun the tests if I can get the server.
> > 
> > Someone else can probably figure it out from this point, but I can't :( 
> > I saw in your test script that you were just counting failues, but not 
> > checking the type. Your test script will have to actually inspect the 
> > response of the set command and the response of the get command and log 
> > any actual errors, as well as anything logged on the server end (run 
> > memcached in the foreground for your tests).
> 
> I would happy take some changes to me script to do better diagnostic 
> checking and reporting if you have any specific ideas. I can check to 
> see if the set failed or if the get failed or succeeded but with 
> different contents. Is here more I can do?

There are a lot of reasons the memcached could be failing, so its
important to find out why.  Try using the 'memcached-tool' in the
'scripts' directory of the memcached distribution.  It will tell you how
the cache is being used.

I don't know if you've read about how memcached works or not, ut
assigning it '-m 1024' doesn't mean that it will store 1 GB of data
before it starts dropping old items.  It means that it will allocate
1024 1MB slabs for caching.  Each 1MB slab will hold 1MB of data within
a certain size range.

For example, if you give '-m 5' and then start filling the cache with
random amounts of data, you might get a 1MB slab for each of:

  Slab 1 :  16 bytes
  Slab 2 :  32 bytes
  Slab 3 :  64 bytes
  Slab 4 : 128 bytes
  Slab 5 : 256 bytes

If your data a lot of things above 16 bytes but below 32 bytes,
memcached may dedicate two 32 byte slabs:

  Slab 1 :  16 bytes
  Slab 2 :  32 bytes
  Slab 3 :  32 bytes
  Slab 4 : 128 bytes
  Slab 5 : 256 bytes

At this point if all new data you cache fits within these slab sizes, if
the cache gets full, it will push out older data.  However, if you now
try to cache something that is 1K, you will get an "out of memory" error
because memcached doesn't have a slab large enough to hold that data and
it won't dump an entire slab to make room for it.

Usually memcached won't create slabs so tightly clustered in size
(though that depends on your usage pattern) but this is one condition
where you'll get a set problem and an out of memory error even if you
have space left in individual slabs.

Garth