Strange memcached behavior on stress test

[dormando]

> Debian 32bit Sarge with a HUGEMEM kernel patch and 6 GB of memory and 1 
> GB configured for memcached. I grew to about 1.5 GB then started 
> throwing the errors.

What were all of the options you used to start memcached?

> The stress test script generated a random sized data packet (1-20KB in 
> size) then does a set and an immediate get and compares the set and and 
> got value, it counts it as an error if they do not match. this was done 
> via a perl script. Sorry, I don't have more specific data, I would have 
> to rerun the tests if I can get the server.

Someone else can probably figure it out from this point, but I can't :( 
I saw in your test script that you were just counting failues, but not 
checking the type. Your test script will have to actually inspect the 
response of the set command and the response of the get command and log 
any actual errors, as well as anything logged on the server end (run 
memcached in the foreground for your tests).

In my mind it could still be a few different things, but more 
information would narrow that down more. I'd also say give the facebook 
branch a try just in case that fixes it.

However even without the facebook patches the memcached instances I run 
hav been chugging along for 6+ months without causing problems, nor 
needing restarts. We might not be pushing the memory limits as hard as 
your test script though.

-Dormando