Memcached crashing on FreeBSD

[Jason Coene]

Hey Guys,

We have a winner - backtrace below, here's what it sent to screen:

memcached in malloc(): error: allocation failed
Abort trap (core dumped)

Thanks,

Jason

--

root at n01> gdb /usr/local/bin/memcached /usr/local/bin/memcached.core
GNU gdb 5.2.1 (FreeBSD)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...
Core was generated by `memcached'.
Program terminated with signal 6, Aborted.
Reading symbols from /lib/libc.so.5...done.
Loaded symbols for /lib/libc.so.5
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x280c5d4f in kill () from /lib/libc.so.5
(gdb) bt
#0  0x280c5d4f in kill () from /lib/libc.so.5
#1  0x280ba7f8 in raise () from /lib/libc.so.5
#2  0x28132f02 in abort () from /lib/libc.so.5
#3  0x2813167e in tcflow () from /lib/libc.so.5
#4  0x28131f1b in tcflow () from /lib/libc.so.5
#5  0x281322d6 in malloc () from /lib/libc.so.5
#6  0x0804b56d in slabs_newslab (id=5) at slabs.c:111
#7  0x0804b605 in slabs_alloc (size=55193) at slabs.c:161
#8  0x0804b9f2 in item_alloc (key=0xbfbfe6f0 "cache_news_overview_2156",
flags=0, exptime=0, nbytes=55133) at items.c:56
#9  0x08049f20 in process_command (c=0x9061400, command=0x2476f000 "set
cache_news_overview_2156 1 1113226764 55131") at memcached.c:513
#10 0x0804a537 in try_read_command (c=0x9061400) at memcached.c:736
#11 0x0804a7bc in drive_machine (c=0x9061400) at memcached.c:836
#12 0x0804ac0d in event_handler (fd=0, which=2, arg=0x9061400) at
memcached.c:1079
#13 0x0804cb78 in event_process_active () at event.c:210
#14 0x0804cd23 in event_loop (flags=0) at event.c:296
#15 0x0804b295 in main (argc=1024, argv=0xbfbfe94c) at memcached.c:1460
#16 0x08049182 in _start ()
(gdb) q

> -----Original Message-----
> From: memcached-bounces at lists.danga.com [mailto:memcached-
> bounces at lists.danga.com] On Behalf Of Jason Coene
> Sent: Saturday, April 09, 2005 1:45 AM
> To: martine at danga.com; 'Brad Fitzpatrick'
> Cc: memcached at lists.danga.com
> Subject: RE: Memcached crashing on FreeBSD
> 
> Thanks guys.
> 
> I had the daemon running as a user who couldn't write to the directory it
> was in, so I don't have a core to work with.  That should be resolved now;
> it takes around 3-4 days for the crash to occur so I'm waiting until then.
> Core file size is unlimited.
> 
> Also, I just found this in screen, the results of the crash inside of the
> restart script:
> 
> Command -u pgsql -p 11212 -m 512 -c 1024
> ------------------------
> memcached in malloc(): error: allocation failed
> Abort trap
> 
> The same output above occurs with every crash.  I'll send a backtrace when
> it crashes again.
> 
> Thanks,
> 
> Jason
> 
> > -----Original Message-----
> > From: Evan Martin [mailto:evan.martin at gmail.com]
> > Sent: Saturday, April 09, 2005 12:12 AM
> > To: Brad Fitzpatrick
> > Cc: Jason Coene; memcached at lists.danga.com
> > Subject: Re: Memcached crashing on FreeBSD
> >
> > You can get backtraces from core files, too.
> >
> > ulimit -c unlimited
> > then run memcached until it crashes, then
> > gdb /path/to/memcached /path/to/core/file
> > then run the "bt" command and send it to the list.
> >
> > On Apr 8, 2005 1:36 PM, Brad Fitzpatrick <brad at danga.com> wrote:
> > > Could you run a debug build and run gdb in screen, then run your
> runner
> > > script in another window, waiting to start it up once the other one
> > > crashes?
> > >
> > > I guess it wouldn't start up until the crashed one (still alive in
> gdb)
> > > closes its listening socket on port 11211.   Is there a gdb command to
> > > make a program close all its sockets on crash?
> > >
> > > Or in gdb just type "run <args>" and then once it starts, immediately
> > type
> > > "bt<enter>" so it'll backgrace on crash.  In another window, run a
> > script
> > > to every second check if memcached isn't responding, and if not, kill
> > gdb
> > > (it already did its backtrace presumably) and then run your normal
> > runner.
> > >
> > > I'm down for other suggestions, though.
> > >
> > >
> > > On Fri, 8 Apr 2005, Jason Coene wrote:
> > >
> > > > Hello,
> > > >
> > > > I'm using memcached in a fairly high traffic environment (but then
> > again,
> > > > who among us aren't!).  I have the memcached server running on
> FreeBSD
> > > > 5.2-RELEASE and 5.3-RELEASE machines, and every day or two the
> service
> > > > suddenly crashes (on both machines).  There are no hardware issues
> > with
> > > > either machine.
> > > >
> > > > >From dmesg:
> > > >
> > > > pid 2408 (memcached), uid 70: exited on signal 6
> > > > pid 4872 (memcached), uid 70: exited on signal 6
> > > > pid 9778 (memcached), uid 70: exited on signal 6
> > > > pid 15861 (memcached), uid 70: exited on signal 6
> > > > pid 24009 (memcached), uid 70: exited on signal 6
> > > > pid 28084 (memcached), uid 70: exited on signal 6
> > > > pid 33100 (memcached), uid 70: exited on signal 6
> > > > pid 45761 (memcached), uid 70: exited on signal 6
> > > > pid 58788 (memcached), uid 70: exited on signal 6
> > > >
> > > > Libevent 0.9 built with:
> > > > ./configure --prefix=/usr/local/
> > > >
> > > > Memcached 1.1.11 built with:
> > > > ./configure --prefix=/usr/local/ --with-libevent=/usr/local/
> > > >
> > > > Memcached is run inside of a restart script (this has been happening
> > for a
> > > > while, just tracking it down now) with commands:
> > > >
> > > > screen -A -m -d -S "Memcached 11211" /usr/local/bin/memcachedrun -u
> > > > memcached -p 11211 -m 1024 -c 1024
> > > >
> > > > "memcachedrun" is just an infinite loop calling
> > /usr/local/bin/memcached $*
> > > >
> > > > I've searched but couldn't find any known issues, has anyone had
> > similar
> > > > problems?  Is there something I can do to debug the issue?
> > > >
> > > > Thanks,
> > > >
> > > > Jason
> > > >
> > > >
> > >