Memory efficiency

[Tobias Lütke]

Hello,

I'd like to ask for advice with the memcached command line parameters.

We store gzipped versions of full html pages in memcached which we
serve out to clients who ask for them "as is" as long as they support
gzip transfer, unpacking them otherwise.

The system works great but i suspect that it utilizes bigger then
average value sizes and the memory efficiency proves to be fairly
poor.

After two days running the bytes stat has ceased to increase. 256mb
have been allocated and 177mb have been used.

There are 68858 items in the cache which means that the average item
size is around 2.6kb.

Lets simulate this with a simple ruby script:

require 'rubygems'
require 'memcache'      # gem install memcache-client
require 'memcache_util' # gem install memcache-client

SERVER = 'localhost:11211'

CACHE = MemCache.new(SERVER)

50_000.times do |i|
  # between 1 and 5 kb of chars
  data = 'd' * (1000 + rand(3000))
  CACHE.set("key:#{i}", data)
end

stats = CACHE.stats[SERVER]

puts 'Unused        : %.2f%% '  % [stats['bytes'].to_f /
stats['limit_maxbytes'].to_f * 100]
puts "max items     : %d " % [stats['curr_items']]
puts "Total cache   : %d bytes" % [stats['limit_maxbytes']]
puts "Total usage   : %d bytes" % [stats['bytes']]
puts "Avg item size : %d bytes" % [stats['bytes'].to_i /
stats['curr_items'].to_i]
puts "Items evicted : %d "      % [stats['total_items'].to_i -
stats['curr_items'].to_i]

This will insert 50k items of our average size in a memcached instance.

Running memcached from the command line without any switches we get
the following results:

~/Code ruby fillmemcached.rb
Unused        : 41.69%
max items     : 11432
Total cache   : 67108864 bytes
Total usage   : 27977981 bytes
Avg item size : 2447 bytes
Items evicted : 38568

Meaning that over 40% of the memory is not used for caching. 11K items
were successfully put into memory, 38568 evicted.

It gets slightly better when changing the factor setting to -f2

~/Code ruby fillmemcached.rb
Unused        : 60.81%
max items     : 16128
Total cache   : 67108864 bytes
Total usage   : 40812063 bytes
Avg item size : 2530 bytes
Items evicted : 33872

we now have 60% unused memory but 16k made it into memory which is
slightly better i suppose.

I tried to play around with the -s switch but it causes my memcached
server to not accept connections anymore.

Any hints on how to make the memory usage better would be much appreciated!

-- 
Tobi
http://shopify.com       - modern e-commerce software
http://typo.leetsoft.com - Open source weblog engine
http://blog.leetsoft.com - Technical weblog