Perlbal crashing on a regular basis

[dormando]

Did you increase the fd limit appropriately? (ulimit -n 64000), ensure 
your OS has a high enough maxfiles limit, etc?

It doesn't actually crash at all? It only ever freezes?

-Dormando

Kevin Olson wrote:
> First of all, thank you to everyone who provided some great ideas yesterday
> for running multiple instances of perlbal.  In the meantime while I'm
> implementing a solution, I've run into some other issues.
> 
> After optimizing our perlbal installation, making sure XS headers were on,
> etc, we're currently handling about 40,000 requests/min only using about 86%
> of the cpu with a small memory footprint.
> 
> But now I'm having problems with perlbal mysteriously ( and relatively
> silently ) freezing/crashing on a regular basis.
> 
> At first I was getting the notorious "too many open files" error before it
> crashed, so I up'd max_connections from 20,000 to 50,000.  Now it'll be
> chugging along for 15-30 minutes, and then it'll just freeze up.  If I set
> PERLBAL_DEBUG to 2, what I see from one min to the next is perlbal accepting
> new connections and sending them to the selector, and then just a bunch of
> messages coming from ClientProxy, telling me that it's closing backends.
> 
> Once it's frozen, it'll either stay that way no matter what I do, or
> sometimes once I remove the connection load, it'll somewhat recover after
> 2-10 minutes.
> 
> Thanks again,
> Kevin
> 
> 
> BTW, here's the main settings from my config file:
> 
> SERVER max_connections = 50000
> XS enable headers
> 
> CREATE SERVICE picserv_bal
>     SET role = reverse_proxy
>     SET pool = picserv_pool
>     SET verify_backend = on
>     SET connect_ahead = 5
>     SET persist_backend = on
>     SET plugins=stats
> ENABLE picserv_bal
>