Memcached crashing on FreeBSD

[Ted Schundler]

Reading this, it seems the issue you guys are looking into is malloc
causing SIGABRT. But shouldn't it not be failing to begin with?

I think your problem is MAXDSIZ. FreeBSD by default limits any program
to a maximum of 512MB data. If you want to use more, add as line like
the following to /boot/loader.conf:

kern.maxdsiz=754974720

(value is in bytes, "G" or "M" suffixes don't work)

The value should be less than your physical memory, and I think it's
prefferable for maxdsize + maxssiz (stack size) to be < physmem

Also, kern.dfldsiz in /boot/loader.conf sets the default limit, and
you may want to bump that as well, or use limits(1) when starting the
program.

Ted

On 4/14/05, Jason Coene <jcoene at gotfrag.com> wrote:
> Bad news, still crashing!
> 
> Command -u pgsql -p 11212 -m 512 -c 1024
> ------------------------
> memcached in malloc(): error: allocation failed
> Abort trap
> 
> User pgsql is uid 70 gid 70, was not a member of wheel (gid 0) group.
> Because of this I don't have a core to backtrace, though I doubt it's much
> different.
> 
> Back to square one, I guess...
> 
> Jason
> 
> > -----Original Message-----
> > From: memcached-bounces at lists.danga.com [mailto:memcached-
> > bounces at lists.danga.com] On Behalf Of Jason Coene
> > Sent: Wednesday, April 13, 2005 8:06 PM
> > To: 'Anatoly Vorobey'; memcached at lists.danga.com
> > Subject: RE: Memcached crashing on FreeBSD
> >
> > Wow - good catch!  I now have memcached running as a non-root user.  The
> > binary is unchanged since the _malloc_options patch... I'll let you know
> > if
> > it goes belly up again.
> >
> > Thanks for all the help!
> >
> > Jason
> >
> > > -----Original Message-----
> > > From: memcached-bounces at lists.danga.com [mailto:memcached-
> > > bounces at lists.danga.com] On Behalf Of Anatoly Vorobey
> > > Sent: Wednesday, April 13, 2005 7:05 PM
> > > To: memcached at lists.danga.com
> > > Subject: Re: Memcached crashing on FreeBSD
> > >
> > > On Wed, Apr 13, 2005 at 06:42:44PM -0400, Jacob Coby wrote:
> > > > Jason Coene wrote:
> > > > >Well guys, some bad news - "ax" didn't fix the crash!
> > > > >
> > > > >Code I added to main():
> > > > >
> > > > >#ifdef __FreeBSD__
> > > > >    _malloc_options = "ax";
> > > > >#endif
> > > > >
> > > > >The ifdef block is definitely getting picked up, I tested with a
> > printf
> > > >
> > > > Maybe an assert() needs to be put in slabs_newslab to double check
> > that
> > > > _malloc_options isn't being reset to the defaults?
> > >
> > > Maybe, but I doubt it; based on reading the source, _malloc_options
> > cannot
> > > matter late in the game, it's only getting parsed during init time.
> > >
> > > > >Backtrace from latest crash:
> > > > >
> > > > >(gdb) bt
> > > > >#0  0x280c5d4f in kill () from /lib/libc.so.5
> > > > >#1  0x280ba7f8 in raise () from /lib/libc.so.5
> > > > >#2  0x28132f02 in abort () from /lib/libc.so.5
> > > > >#3  0x2813167e in tcflow () from /lib/libc.so.5
> > > > >#4  0x28131f1b in tcflow () from /lib/libc.so.5
> > > >
> > > > I'm very naive with the *BSDs, but #3 and #4 confuse me a little bit.
> > I
> > > > haven't been able to find a version of malloc() in the FreeBSD CVS
> > that
> > > > calls tcflow().  The version that I'm looking at calls pubrealloc(),
> > > > which never calls tcflow().  It'd be nice if the bt showed the params
> > to
> > > > the libc calls..
> > >
> > > Ah, well, this is probably due to the libc.so.5 not being a debug
> > version
> > > (and why would it be), so it doesn't have debug information for internal
> > > functions. All gdb has to go on are the addresses of the exported
> > > functions, and #3 and #4 are just some internal functions used by malloc
> > > that happen to lie closest to tcflow()'s address in the code segment,
> > > among all the exported addresses gdb has.
> > >
> > > In fact, we can reconstruct the flow, using the fact that it's FreeBSD
> > > 5.2-RELEASE, provided by Jason. The relevant source file is
> > >
> > >
> > http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libc/stdlib/malloc.c?rev=1.8
> > > 4.2.1&content-type=text/x-cvsweb-markup&only_with_tag=RELENG_5_2
> > >
> > > Jason's getting "error: allocation failed", which is only called from
> > > inside imalloc() in that file. imalloc() calls wrterror() - these are
> > our
> > > #3 and #4 - and wrterror() calls abort(). imalloc() in its turn is
> > called
> > > directly from malloc().
> > >
> > > But why does imalloc() abort, even though we told it not to? It looks at
> > > the internal variable malloc_abort as a guide of whether to abort. The
> > > variable is initialised from all three kinds of malloc options (symlink,
> > > environment variable and _malloc_options) in malloc_init(). It
> > corresponds
> > > directly to the option 'a':
> > >
> > > case 'a': malloc_abort   = 0; break;
> > >
> > > But look at what happens right after that:
> > >
> > >     /*
> > >      * Sensitive processes, somewhat arbitrarily defined here as setuid,
> > >      * setgid, root and wheel cannot afford to have malloc mistakes.
> > >      */
> > >     if (issetugid() || getuid() == 0 || getgid() == 0)
> > >         malloc_abort = 1;
> > >
> > > Somewhat arbitrarily, and completely undocumented.
> > >
> > > So, Jason, you need to try and run memcached as a non-root user *and*
> > with
> > > the _malloc_options change. If even that will fail with the same
> > problem,
> > > we'll be back at square one.
> > >
> > > --
> > > avva
> > > "There's nothing simply good, nor ill alone" -- John Donne
> >
> 
>