Busy loop with memcached 1.2.2

[Paul Lindner]

On Mon, Aug 20, 2007 at 04:30:18PM +0300, Kari Lavikka wrote:
> Hi,
> 
> I just upgraded (and downgraded) a few of our memcached servers to 1.2.2 
> from 1.2.1. After memory usage reached about 1.4GB every of the memcached 
> instances went to busy loop (99.9% cpu and didn't accept any connections).

Okay.  1.2.1 -> 1.2.2 had quite a few changes. So it will be hard to
pin it on any specific thing.  Did you enable threads on your build?

> We have debian linux on 64bit dual opterons with 8GB of memory. Each 
> server runs two instances. Libevent is: libevent-1.1a.so.1.0.2

That's a pretty old libevent, 1.1b has some bugfixes

> Command line arguments were:
> memcached -m 3500 -c 8192 -d -P /var/run/memcached2.pid -p 11212 -u 
> memcached

You might try removing the -d flag if you plan to use the -P flag at
the same time.  There was a bug that surfaced with that in 1.2 on
x86_64 due to some odd interaction between libevent and open file
descriptors.

Thanks
Paul

> Some stack traces:
> 
> 0x0000000000407051 in assoc_find (key=0xc4a1f4 "image_36411271", nkey=14) 
> at assoc.c:503
> 503             if ((nkey == it->nkey) &&
> (gdb) bt
> #0  0x0000000000407051 in assoc_find (key=0xc4a1f4 "image_36411271", 
> nkey=14) at assoc.c:503
> #1  0x0000000000406a7e in do_item_get_notedeleted (key=0xc4a200 "71", 
> nkey=3707308320, delete_locked=0x0) at items.c:370
> #2  0x0000000000404072 in event_handler (fd=12886528, which=2336, 
> arg=0x9a0940) at memcached.c:1086
> #3  0x00002aaaaacdc0e2 in event_base_loop () from 
> /usr/lib/libevent-1.1a.so.1
> #4  0x00000000004058dc in main (argc=13393, argv=0x7fffffaec028) at 
> memcached.c:2742
> 
> 
> 0x000000000040704e in assoc_find (key=0x1ce8f74 
> "relevant_CAMP_FREQ_1490_312_4_2007082014", nkey=40) at assoc.c:503
> 503     assoc.c: No such file or directory.
> (gdb) bt
> #0  0x000000000040704e in assoc_find (key=0x1ce8f74 
> "relevant_CAMP_FREQ_1490_312_4_2007082014", nkey=40) at assoc.c:503
> #1  0x0000000000406a7e in do_item_get_notedeleted (key=0x1ce8f98 "2014", 
> nkey=948781420, delete_locked=0x0) at items.c:370
> #2  0x0000000000404072 in event_handler (fd=30314392, which=16748, 
> arg=0xdddf10) at memcached.c:1086
> #3  0x00002aaaaacdc0e2 in event_base_loop () from 
> /usr/lib/libevent-1.1a.so.1
> #4  0x00000000004058dc in main (argc=6838, argv=0x7fffff944ed8) at 
> memcached.c:2742
> 
> 
>     |\__/|
>     ( oo )    Kari Lavikka - tuner at bdb.fi - (050) 380 3808
> __ooO(  )Ooo_______ _____ ___ _ _  _   _    _      _                  _
>       ""
> 

-- 
Paul Lindner        ||||| | | | |  |  |  |   |   |
lindner at inuus.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.danga.com/pipermail/memcached/attachments/20070821/fb8eb4fb/attachment.pgp