Perlbal and Last.fm

[Russ Garrett]

Hi, a week or two on, here are a couple of issues we've found with
Perlbal (we are using the CVS, I lied when I said the tarball - I was
thinking of memcached...):

Firstly, we seem to be getting a significant degree of memory leakage
with perlbal - before I restarted today it was using 1.3GB of memory,
which is never a good thing. It's running ulimited now, so we shall see
if this improves things.


Secondly, and I don't know if this is related, we're getting some errors
from perlbal:

epoll() returned fd 77 w/ state 1 for which we have no mapping.
removing.
epoll() returned fd 30 w/ state 1 for which we have no mapping.
removing.
Use of uninitialized value in split
at /usr/local/share/perl/5.8.4/Net/Netmask.pm line 242.
Use of uninitialized value in bitwise and (&)
at /usr/local/share/perl/5.8.4/Net/Netmask.pm line 404.

As you can see we're using Perl 5.8.4 (the current version in Debian
Sarge).


Lastly, and I think this is probably specific to our setup and the fact
that PHP is shit, we sometimes get complete dropouts where the CPU usage
on the web heads drops to almost 0, and perlbal starts queueing
everything while it connects some more backends. Maybe our setup needs
more tweaking

How is MaxClients set up on LiveJournal? I initially tried it down as
low as 10, but it seemed to be having trouble saturating the servers at
that level (probably our PHP issues again). It's now up to 60, and that
seems a lot better, however that's increasing the amount of concurrency
on our DB servers, which isn't necessarily a good thing...

Cheers,

-- 
Russ Garrett                            Last.fm Limited
russ at last.fm                             http://last.fm