Perlbal crashing on a regular basis
dormando at rydia.net
Wed Jan 30 19:21:52 UTC 2008
Did you increase the fd limit appropriately? (ulimit -n 64000), ensure
your OS has a high enough maxfiles limit, etc?
It doesn't actually crash at all? It only ever freezes?
Kevin Olson wrote:
> First of all, thank you to everyone who provided some great ideas yesterday
> for running multiple instances of perlbal. In the meantime while I'm
> implementing a solution, I've run into some other issues.
> After optimizing our perlbal installation, making sure XS headers were on,
> etc, we're currently handling about 40,000 requests/min only using about 86%
> of the cpu with a small memory footprint.
> But now I'm having problems with perlbal mysteriously ( and relatively
> silently ) freezing/crashing on a regular basis.
> At first I was getting the notorious "too many open files" error before it
> crashed, so I up'd max_connections from 20,000 to 50,000. Now it'll be
> chugging along for 15-30 minutes, and then it'll just freeze up. If I set
> PERLBAL_DEBUG to 2, what I see from one min to the next is perlbal accepting
> new connections and sending them to the selector, and then just a bunch of
> messages coming from ClientProxy, telling me that it's closing backends.
> Once it's frozen, it'll either stay that way no matter what I do, or
> sometimes once I remove the connection load, it'll somewhat recover after
> 2-10 minutes.
> Thanks again,
> BTW, here's the main settings from my config file:
> SERVER max_connections = 50000
> XS enable headers
> CREATE SERVICE picserv_bal
> SET role = reverse_proxy
> SET pool = picserv_pool
> SET verify_backend = on
> SET connect_ahead = 5
> SET persist_backend = on
> SET plugins=stats
> ENABLE picserv_bal
More information about the perlbal