Perlbal crashing on a regular basis

[Kevin Olson]

First of all, thank you to everyone who provided some great ideas yesterday
for running multiple instances of perlbal.  In the meantime while I'm
implementing a solution, I've run into some other issues.

After optimizing our perlbal installation, making sure XS headers were on,
etc, we're currently handling about 40,000 requests/min only using about 86%
of the cpu with a small memory footprint.

But now I'm having problems with perlbal mysteriously ( and relatively
silently ) freezing/crashing on a regular basis.

At first I was getting the notorious "too many open files" error before it
crashed, so I up'd max_connections from 20,000 to 50,000.  Now it'll be
chugging along for 15-30 minutes, and then it'll just freeze up.  If I set
PERLBAL_DEBUG to 2, what I see from one min to the next is perlbal accepting
new connections and sending them to the selector, and then just a bunch of
messages coming from ClientProxy, telling me that it's closing backends.

Once it's frozen, it'll either stay that way no matter what I do, or
sometimes once I remove the connection load, it'll somewhat recover after
2-10 minutes.

Thanks again,
Kevin


BTW, here's the main settings from my config file:

SERVER max_connections = 50000
XS enable headers

CREATE SERVICE picserv_bal
    SET role = reverse_proxy
    SET pool = picserv_pool
    SET verify_backend = on
    SET connect_ahead = 5
    SET persist_backend = on
    SET plugins=stats
ENABLE picserv_bal