memcached-1.1.12 does infinite loop

[Andreas Vierengel]

I can confirm this behaviour. I have recently tested get_multi with a
variable amount of keys. On my 2.8 GHz Xeon machine it took 0.8s so
process a get_multi with 10.000 keys. During this time, nothing else is
done by memcached. ( I have also done 500.000 keys and this stalls
everything for minutes ).
About 1% of my requests do get_multi with about 1000 keys, so I'm going
to limit one get_multi to about 500 keys, too.

After looking at the code, I think the granularity of an event is one
command. If the command is a 500.000-keys-get_multi memcached is forced
to satisfy this request before continuing with other work.

If this is intended by the authors, it should be at least a note in the
docs and/or the client should wrap this to several get_multis ?!?
I personally can live with this :)

Another idea is to rewrite portions of the memcached-code, so that
event-handling is not only done for async network I/O, but also within a
get_multi-request. Maybe in the event of a writeable socket, the
get_multi fetches more key/values from this request until the socket
write-buffer is full. The "state-machine" must be enhanced for this, I
think. Buffering the whole response, like it is done at the moment, will
also not be needed.
I don't think this would be against the whole O(1)-conecpt.

--Andy


Am Montag, den 23.05.2005, 12:26 +0200 schrieb Elizabeth Mattijsen:
> At 4:56 PM -0700 5/22/05, Brad Fitzpatrick wrote:
> >A repeatable test case would be great.  Has anybody else on the list ran
> >into this?
> 
> Maybe.  I've seen a memcached daemon spin for several minutes before 
> coming back to live.  I've attributed it to an excessive amount of 
> objects requested in a get_multi.  Since then I'm limiting to 512 
> objects per get_multi and I haven't seen the problem reoccur.
> 
> This was at significanly less set cmds (max about 10/sec) and get 
> cmds (max about 15/sec).
> 
> 
> Liz
> ==================================
> >On Sun, 22 May 2005, Andreas Vierengel wrote:
> >
> >>  hi,
> >>
> >>  I was running two memcached servers for 2 weeks successfully. Each
> >>  server get's about max 3000-6000 set-cmds per second and about max
> >>  300-600 get-cmds per second and nearly max 1200 persistent-connections.
> >>  Today both server suddenly started to infinite loop, sucking 99% cpu.
> >>  I tried to strace them, but strace prints nothing, so I think it's some
> >>  sort of userland-infinite-loop. Killing and restarting brought it back
> >>  to work again.
> >>  top says that each process had 2GB virt, 1.7GB res and 580 share. Memory
> >>  consumption was constantly growing. The stats said that there were about
> >>  4 billion set-cmds. About 700.000 keys are active. the values range from
> >>  280-350 bytes.
> >>
> >>  startup is:
> >>  ulimit -n 30000
> >>  memcached -u nobody -p 2000 -m 1900 -c 25000
> >>
> >>  version is 1.1.12
> >>
> >>  system is debian sarge with vanilla-kernel 2.6.11, 2GB RAM,
> >>  dual-P4-Xeon.
> >>  On my clients I get no errors, but every request was answered with undef
> >>  ( I'm using perl ), pushing my database hard :)
> >>
> >>  I will try several tests the next days to make it reproducible.
> >>
> >>  --Andy
> >>
> >>
> 
>