Perlbal crashing on a regular basis

dormando dormando at rydia.net
Wed Jan 30 19:31:28 UTC 2008


You running perlbal as root? :)

That's not always worked for me; which is why I suggested trying the 
manual increase to something significantly higher than max_connections 
is set to.

Shouldn't crash if max_connections is set too low either. You're on the 
latest available release, right?

-Dormando

Kevin Olson wrote:
> I was under the impression that that’s what max_connections did using 
> BSD::Resource.
> Lib/Perlbal.pm:
> 
>  if ($key eq "max_connections") {
>         return $mc->err('This command is not available unless 
> BSD::Resource is installed') unless $Perlbal::BSD_RESOURCE_AVAILABLE;
>         return $mc->err("Expected numeric parameter") unless $val =~ 
> /^-?\d+$/;
>         my $rv = 
> BSD::Resource::setrlimit(BSD::Resource::RLIMIT_NOFILE(), $val, $val);
>         unless (defined $rv && $rv) {
>             if ($> == 0) {
>                 $mc->err("Unable to set limit.");
>             } else {
>                 $mc->err("Need to be root to increase max connections.");
>             }
>         }
>         return $mc->ok;
> 
> Yep, only seems to crash when max_connections is set too low.
> 
> 
> On 1/30/08 11:21 AM, "dormando" <dormando at rydia.net> wrote:
> 
>     Did you increase the fd limit appropriately? (ulimit -n 64000), ensure
>     your OS has a high enough maxfiles limit, etc?
> 
>     It doesn't actually crash at all? It only ever freezes?
> 
>     -Dormando
> 
>     Kevin Olson wrote:
>     >  First of all, thank you to everyone who provided some great ideas
>     yesterday
>     >  for running multiple instances of perlbal.  In the meantime while I'm
>     >  implementing a solution, I've run into some other issues.
>     >
>     >  After optimizing our perlbal installation, making sure XS headers
>     were on,
>     >  etc, we're currently handling about 40,000 requests/min only using
>     about 86%
>     >  of the cpu with a small memory footprint.
>     >
>     >  But now I'm having problems with perlbal mysteriously ( and relatively
>     >  silently ) freezing/crashing on a regular basis.
>     >
>     >  At first I was getting the notorious "too many open files" error
>     before it
>     >  crashed, so I up'd max_connections from 20,000 to 50,000.  Now
>     it'll be
>     >  chugging along for 15-30 minutes, and then it'll just freeze up.
>      If I set
>     >  PERLBAL_DEBUG to 2, what I see from one min to the next is perlbal
>     accepting
>     >  new connections and sending them to the selector, and then just a
>     bunch of
>     >  messages coming from ClientProxy, telling me that it's closing
>     backends.
>     >
>     >  Once it's frozen, it'll either stay that way no matter what I do, or
>     >  sometimes once I remove the connection load, it'll somewhat
>     recover after
>     >  2-10 minutes.
>     >
>     >  Thanks again,
>     >  Kevin
>     >
>     >
>     >  BTW, here's the main settings from my config file:
>     >
>     >  SERVER max_connections = 50000
>     >  XS enable headers
>     >
>     >  CREATE SERVICE picserv_bal
>     >      SET role = reverse_proxy
>     >      SET pool = picserv_pool
>     >      SET verify_backend = on
>     >      SET connect_ahead = 5
>     >      SET persist_backend = on
>     >      SET plugins=stats
>     >  ENABLE picserv_bal
>     >
> 
> 
> 



More information about the perlbal mailing list