Linux sometimes kernel panics when starting memcached... (eek)

[Brad Fitzpatrick]

I'm wondering if we should ditch epoll and libevent and do our own
abstraction, choosing between either kqueue on FreeBSD or real-time
signals on Linux.

Avva?


On Tue, 2 Sep 2003, Justin Matlock wrote:

> As for not connecting to it...  it will just sit there, happily taking up
> memory and no CPU when nothing connects to it.   I can load the webserver
> down hard (I used the three other boxes to use ab to slam it), and the
> server does fine... but...
>
> I can load as much data into memcached as I want into it.. but the second I
> retrieve more than 2 chunks, it explodes.
>
> This last crash was pretty cool though... the speaker started making this
> bizarre screeching noise and wouldn't stop until I yanked the cord out of
> the back.  The power switch wouldn't even turn it off.  :-)
>
> Definitely an epoll problem, but I can't figure out what changed in 2.4.22
> that would have broken it this badly....  Anyway, it's not really a big
> issue for me, I'll just drop back to 2.4.21...
>
>
> ----- Original Message -----
> From: "Brad Fitzpatrick" <brad@danga.com>
> To: <memcached@lists.danga.com>
> Sent: Tuesday, September 02, 2003 4:47 PM
> Subject: Re: Linux sometimes kernel panics when starting memcached... (eek)
>
>
> > This isn't too surprising, after already coming to the conclusion that the
> > Linux epoll patches are buggy.  (we've been fighting a libevent
> > and/or epoll bug for the past few weeks that rarely manifests itself, and
> > usually only on 2.4.21, we think?)
> >
> > Some or all of us should start bugging the epoll guy.  Anybody feel like
> > preparing a binary of memcached and test perl script we could give him to
> > segfault his system?  (after which he could debug...)
> >
> > - Brad
> >
> >
> > On Tue, 2 Sep 2003, Justin Matlock wrote:
> >
> > > Ah.. yes. I should have mentioned that.
> > >
> > > Clients are connecting to the memcached I started.  It runs for about 3
> > > minutes before the kernel panic.  The data is usually in 1-2k chunks,
> and
> > > I'm not storing very much in there ... maybe 5-10 mb at the most has
> been
> > > stored before it dies.   I have memcached set to use 512mb RAM.
> > >
> > >
> > > ----- Original Message -----
> > > From: "Anatoly Vorobey" <mellon@pobox.com>
> > > To: <memcached@lists.danga.com>
> > > Sent: Tuesday, September 02, 2003 4:18 PM
> > > Subject: Re: Linux sometimes kernel panics when starting memcached...
> (eek)
> > >
> > >
> > > > Justin,
> > > >
> > > > You wrote on Tue, Sep 02, 2003 at 04:12:35PM -0400:
> > > > > This is the strangest thing I've seen...
> > > > >
> > > > > I just upgraded two of my webservers to Linux 2.4.22, virgin, which
> I
> > > then patched with the ipvs and the epoll patches; both applied cleanly,
> > > without rejections.  The servers work fine, until I start up
> memcached --
> > > and every single time, the machines will kernel panic (aiee!) within 3
> > > minutes.  If I don't start memcached, the machine runs fine for hours --
> > > until I get bored waiting for it to kernel panic, and I start
> memcached..
> > > *boom*.
> > > > >
> > > > > I have no idea how memcached could be causing this, unless there's
> > > something funky about the epoll patch (that's written for 2.4.21, but it
> > > merges clean into 2.4.22).
> > > >
> > > > Do you use any client to connect to memcached that you started, or is
> > > > merely starting the program, without any connections, sufficient to
> > > > crash the server after a while?
> > > >
> > > > --
> > > > avva
> > > >
> > >
> > >
> >
>
>