Perlbal and Last.fm

[Russ Garrett]

On Thu, 2005-01-27 at 22:13 -0800, Mark Smith wrote:
> Interesting!  We don't have (much) memory leakage with Perlbal.  It's
> been up for about a day or two (since the last restart) and is using
> only 65MB of memory each, doing 500-600 requests per second (each)...
> 
> I'd be interested in knowing more about your usage pattern--what kind of
> traffic are you sending at it, and what are you having it do?  (The most
> informative would be to see your config file.)

Well suffice to say our usage pattern is nowhere near LJ's - we just
have one instance of perlbal balancing the web traffic for both
www.audioscrobbler.com and www.last.fm - I'm guessing that's probably
only 50-100 requests per second at the moment (but I have no idea
because we're not logging anything currently). 

The interesting load (the HTTP song submissions for Audioscrobbler,
which are another 50ish hits per second) doesn't go through perlbal at
the moment, it's just handled by a single instance of tomcat.

Our config is basically persistent backends with verify_backend and
random balancing, we're also using the fastq stuff which works quite
nicely.

As I mentioned I'd be more inclined to blame perl or a perl module, but
I don't know.

> We get those two.

OK, I suspected they were harmless.

> > Lastly, and I think this is probably specific to our setup and the fact
> > that PHP is shit, we sometimes get complete dropouts where the CPU usage
> > on the web heads drops to almost 0, and perlbal starts queueing
> > everything while it connects some more backends. Maybe our setup needs
> > more tweaking
> 
> When did this start happening?  There was a bug in Perlbal for about 48
> hours over the past two or three days (it was fixed today) that caused
> it to mishandle backend connections, sometimes causing long queues
> without there being any real traffic...

Nope it wasn't that, since our CVS copy is about 2 weeks old... Like I
said I'm guessing it's something to do with our setup, but it's quite
odd that all the servers drop out at the same time. I'll try and grab
some more stats next time it happens.

-- 
Russ Garrett                            Last.fm Limited
russ at last.fm                             http://last.fm