SERVER_ERROR out of memory on small puts to memcache.

[Matt Knox]

I consistently get an out of memory error when performing  < 44 byte
(key+data) puts to memcache, but when 'put'-ing the same key with data that
is long enough to exceed 44 chars key + data  , I succeed.  I observe this
behavior using both the ruby client (memcache-client 1.5) and the python one
(python-memcached), although the python client seems to break at 59 chars,
rather than 44.

I see something vaguely similar on the lists in early 2006:
http://www.nabble.com/%22SERVER_ERROR-out-of-memory%22-when-changing-cache-entry-pattern-t4346764.html

but my error is consistent, where theirs seems to have been sporadic.



I'm running memcached 1.2.2 with the following options:

memcached -d -p 11211 -u nobody -c 1024 -m 3072

The offending servers are recent CentOS running 3G of cache each on 32-bit
boxes with 4G ram-if it matters, I can get details about these.



Very detailed:

Here is a ruby console session,

>> s = "1234567890" * 100
=> "1234567890.....{long string elided for brevity.}"
>> s.first(10)
=> "1234567890"
>> CACHE.set('foo', s.first(1))
MemCache::MemCacheError: out of memory
        from /usr/lib/ruby/gems/1.8/gems/memcache-client-1.5.0/lib/memcache.rb:307:in
`set'
        from (irb):16
        from :0
>> CACHE.set('foo', s.first(50))
=> nil
>> CACHE.get('foo')
=> "12345678901234567890123456789012345678901234567890"
>> CACHE.set('foo', s.first(30))
MemCache::MemCacheError: out of memory
        from /usr/lib/ruby/gems/1.8/gems/memcache-client-1.5.0/lib/memcache.rb:307:in
`set'
        from (irb):31
        from :0
>> CACHE.set('foo', s.first(40))
=> nil
>> CACHE.get('foo')
=> "1234567890123456789012345678901234567890"
>> CACHE.set('foo', s.first(32))
MemCache::MemCacheError: out of memory
        from /usr/lib/ruby/gems/1.8/gems/memcache-
client-1.5.0/lib/memcache.rb:307:in
`set'
        from (irb):34
        from :0
>> CACHE.set('foo', s.first(33))
=> nil
>> CACHE.get('foo')
=> "123456789012345678901234567890123"
>> CACHE.get('foobar')
=> nil
>> CACHE.set('foobar', s.first(32))
=> nil
>> CACHE.get('foobar')
=> "12345678901234567890123456789012"
>> CACHE.set('foobar', s.first(30))
=> nil
>> CACHE.get('foobar')
=> "123456789012345678901234567890"
>> CACHE.set('foobar', s.first(29))
MemCache::MemCacheError: out of memory
        from /usr/lib/ruby/gems/1.8/gems/memcache-client-1.5.0/lib/memcache.rb:307:in
`set'
        from (irb):47
        from :0


So it looks like anything under 36 bytes (key + data) has a problem. The
client I use prefixes keys with an 8-char string, so that's actually 44
chars key + data

Using python-memcached, I got substantially the same results, only the break
happened at 59 chars key + data.

>>> s = "1234567890" * 10
>>> mc.set("key", s[0:56])
1
>>> mc.get("key")
'12345678901234567890123456789012345678901234567890123456'

>>> mc.set("key", s[0:55])
MemCached: while expecting 'STORED', got unexpected response 'SERVER_ERROR
out of memory'
1
>>> mc.get("key")
'12345678901234567890123456789012345678901234567890123456'
>>> mc.set("k", s[0:56])
MemCached: while expecting 'STORED', got unexpected response 'SERVER_ERROR
out of memory'
1
>>> mc.set("k", s[0:58])
1
>>> mc.get("k")
'1234567890123456789012345678901234567890123456789012345678'



If it matters, here is the response from stats:

>> CACHE.stats
=> {"x.x.x.x:11211"=>{"bytes"=>2678831108, "pid"=>2613,
"connection_structures"=>47, "threads"=>4, "evictions"=>7397,
"time"=>1191948397, "pointer_size"=>32, "limit_maxbytes"=>3221225472,
"cmd_get"=>54170155, "version"=>"1.2.2", "bytes_written"=>37964380497,
"cmd_set"=>25110788, "get_misses"=>11659284, "total_connections"=>2041765,
"curr_connections"=>43, "curr_items"=>4374286, "uptime"=>2334717,
"get_hits"=>42510871, "total_items"=>25110788, "rusage_system"=>2039.975876,
"rusage_user"=>612.713853, "bytes_read"=>75534170186},
"x.x.x.y:11211"=>{"bytes"=>2678220092,
"pid"=>2516, "connection_structures"=>47, "threads"=>4, "evictions"=>10624,
"time"=>1191951946, "pointer_size"=>32, "limit_maxbytes"=>3221225472,
"cmd_get"=>56914534, "version"=>"1.2.2", "bytes_written"=>34812915115,
"cmd_set"=>25042621, "get_misses"=>11378051, "total_connections"=>2041833,
"curr_connections"=>43, "curr_items"=>4542475, "uptime"=>2334736,
"get_hits"=>45536483, "total_items"=>25042621, "rusage_system"=>2115.619376,
"rusage_user"=>631.239037, "bytes_read"=>75855048310}}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/memcached/attachments/20071009/9f2905e2/attachment.html