Strange memcached behavior on stress test

[Stephen Woodbridge]

dormando wrote:
> 
>> Debian 32bit Sarge with a HUGEMEM kernel patch and 6 GB of memory and 
>> 1 GB configured for memcached. I grew to about 1.5 GB then started 
>> throwing the errors.
> 
> What were all of the options you used to start memcached?

/usr/bin/memcached -m 1024 -p 11211 -u root -l 10.14.12.101

>> The stress test script generated a random sized data packet (1-20KB in 
>> size) then does a set and an immediate get and compares the set and 
>> and got value, it counts it as an error if they do not match. this was 
>> done via a perl script. Sorry, I don't have more specific data, I 
>> would have to rerun the tests if I can get the server.
> 
> Someone else can probably figure it out from this point, but I can't :( 
> I saw in your test script that you were just counting failues, but not 
> checking the type. Your test script will have to actually inspect the 
> response of the set command and the response of the get command and log 
> any actual errors, as well as anything logged on the server end (run 
> memcached in the foreground for your tests).

I would happy take some changes to me script to do better diagnostic 
checking and reporting if you have any specific ideas. I can check to 
see if the set failed or if the get failed or succeeded but with 
different contents. Is here more I can do?

> In my mind it could still be a few different things, but more 
> information would narrow that down more. I'd also say give the facebook 
> branch a try just in case that fixes it.
> 
> However even without the facebook patches the memcached instances I run 
> hav been chugging along for 6+ months without causing problems, nor 
> needing restarts. We might not be pushing the memory limits as hard as 
> your test script though.

I may need to wait for my sysadmin to get back from his honeymoon, 
before I can get patches integrated and deployed to a server for 
additional testing.

-Steve