Memory efficiency

[Tobias Lütke]

Interesting that there were no replies to this issue yet.

I guess the issue is not well understood or did i just completely miss
the mark here?
Is our usage of memcached very out of the ordinary?



On 7/24/07, Tobias Lütke <tobias.luetke at gmail.com> wrote:
> Hello,
>
> I'd like to ask for advice with the memcached command line parameters.
>
> We store gzipped versions of full html pages in memcached which we
> serve out to clients who ask for them "as is" as long as they support
> gzip transfer, unpacking them otherwise.
>
> The system works great but i suspect that it utilizes bigger then
> average value sizes and the memory efficiency proves to be fairly
> poor.
>
> After two days running the bytes stat has ceased to increase. 256mb
> have been allocated and 177mb have been used.
>
> There are 68858 items in the cache which means that the average item
> size is around 2.6kb.
>
> Lets simulate this with a simple ruby script:
>
> require 'rubygems'
> require 'memcache'      # gem install memcache-client
> require 'memcache_util' # gem install memcache-client
>
> SERVER = 'localhost:11211'
>
> CACHE = MemCache.new(SERVER)
>
> 50_000.times do |i|
>   # between 1 and 5 kb of chars
>   data = 'd' * (1000 + rand(3000))
>   CACHE.set("key:#{i}", data)
> end
>
> stats = CACHE.stats[SERVER]
>
> puts 'Unused        : %.2f%% '  % [stats['bytes'].to_f /
> stats['limit_maxbytes'].to_f * 100]
> puts "max items     : %d " % [stats['curr_items']]
> puts "Total cache   : %d bytes" % [stats['limit_maxbytes']]
> puts "Total usage   : %d bytes" % [stats['bytes']]
> puts "Avg item size : %d bytes" % [stats['bytes'].to_i /
> stats['curr_items'].to_i]
> puts "Items evicted : %d "      % [stats['total_items'].to_i -
> stats['curr_items'].to_i]
>
> This will insert 50k items of our average size in a memcached instance.
>
> Running memcached from the command line without any switches we get
> the following results:
>
> ~/Code ruby fillmemcached.rb
> Unused        : 41.69%
> max items     : 11432
> Total cache   : 67108864 bytes
> Total usage   : 27977981 bytes
> Avg item size : 2447 bytes
> Items evicted : 38568
>
> Meaning that over 40% of the memory is not used for caching. 11K items
> were successfully put into memory, 38568 evicted.
>
> It gets slightly better when changing the factor setting to -f2
>
> ~/Code ruby fillmemcached.rb
> Unused        : 60.81%
> max items     : 16128
> Total cache   : 67108864 bytes
> Total usage   : 40812063 bytes
> Avg item size : 2530 bytes
> Items evicted : 33872
>
> we now have 60% unused memory but 16k made it into memory which is
> slightly better i suppose.
>
> I tried to play around with the -s switch but it causes my memcached
> server to not accept connections anymore.
>
> Any hints on how to make the memory usage better would be much appreciated!
>
> --
> Tobi
> http://shopify.com       - modern e-commerce software
> http://typo.leetsoft.com - Open source weblog engine
> http://blog.leetsoft.com - Technical weblog
>


-- 
Tobi
http://shopify.com       - modern e-commerce software
http://typo.leetsoft.com - Open source weblog engine
http://blog.leetsoft.com - Technical weblog