memcached not using allocated memory

[Lewis, Bill]

Steve,

I'm going with the default 1.25 growth as it provides this range:

slab class  24: chunk size  14200 perslab    73
slab class  25: chunk size  17752 perslab    59
slab class  26: chunk size  22192 perslab    47

I assumed that the "bytes" stat would contain the count for allocation for each chunk(17752), and not the actual amount of the chunk used, but that might not be the case.  

If the values are 16384 bytes, they should use the 17752 byte chunk and occupy 92% of it.  I'm not sure about the space for the keys/flags but the keys are all between 10 and 15 bytes.  I've tried tweaking the -f option and the default has provided the best result to date.  Any suggestions. thanx.

-bill


-----Original Message-----
From: Steven Grimm [mailto:sgrimm at facebook.com]
Sent: Fri 4/20/2007 12:28 AM
To: Marcus Bointon
Cc: Lewis, Bill; memcached at lists.danga.com
Subject: Re: memcached not using allocated memory
 
Marcus Bointon wrote:
> I'm not familiar with the internals, but I believe that memory chunks 
> are allocated in power-of-2 sized chunks. 16394 is 10 bytes over 
> 16384, which is 2^14, so your 16394 bytes will be stored in a 32768 
> size chunk (2^15), hence the wastage. It's just unfortunate that your 
> data is that size - it couldn't get much less efficient! That's my 
> theory anyway, could be completely wrong of course!

That's somewhat plausible, although modern versions use a power of 1.25 
to determine the chunk sizes.

It's easy to test that theory: there's a command-line option (-f) that 
lets you set the factor that's used. Try reducing it to 1.05 or some 
similarly small value and you should get a chunk size much closer to 
your data size.

Also, if that's what's going on, and all the objects in the cache are 
truly fixed-size, you can use another command line option to tune the 
smallest chunk size. It is perfectly fine to set that to the exact value 
needed to hold your fixed-size objects (remember that you need to leave 
room for the key as well, plus some space for the flags value; should be 
pretty easy to experimentally determine the optimal size.) Then there 
will be no wasted memory in your slabs at all.

-Steve