memcached-1.1.12 does infinite loop

[Andreas Vierengel]

Am Sonntag, den 22.05.2005, 23:57 +0200 schrieb Andreas Vierengel:
> hi,
> 
> I was running two memcached servers for 2 weeks successfully. Each
> server get's about max 3000-6000 set-cmds per second and about max
> 300-600 get-cmds per second and nearly max 1200 persistent-connections.
> Today both server suddenly started to infinite loop, sucking 99% cpu.
> I tried to strace them, but strace prints nothing, so I think it's some
> sort of userland-infinite-loop. Killing and restarting brought it back
> to work again.
> top says that each process had 2GB virt, 1.7GB res and 580 share. Memory
> consumption was constantly growing. The stats said that there were about
> 4 billion set-cmds. About 700.000 keys are active. the values range from
> 280-350 bytes.
> 
> startup is:
> ulimit -n 30000
> memcached -u nobody -p 2000 -m 1900 -c 25000
> 
> version is 1.1.12
> 
> system is debian sarge with vanilla-kernel 2.6.11, 2GB RAM,
> dual-P4-Xeon.
> On my clients I get no errors, but every request was answered with undef
> ( I'm using perl ), pushing my database hard :)
> 
> I will try several tests the next days to make it reproducible.
> 
> --Andy
> 

on my testsystem i was not able to reproduce the problem again :(
memcached-process sticked with 1900mb resident size. virtual size was
slightly higher, but I noticed a slightly use of swap by the kernel,
although nothing else is consuming memory on the system. Maybe it's the
strategy of the kernel-vm-subsystem, that I hit.
I don't know why I get 2GB Virtual and 1.7GB resident yesterday...
I will try further to reproduce it, otherwise i will let my
production-system run in this weird state again and take a gdb
backtrace.

Another question: what is the best method to determine if it is time to
add another memcached-server to the pool ?

Thanks for your comments !

--Andy