Running multiple perlbal instances

[dormando]

Kevin Olson wrote:
> Does anyone have some advice on running multiple perlbal instances on one
> multi-cpu box?  We're getting to the point where one perlbal instance is
> handling 30,000-45,000 connections/min and load is hitting 1.00.
> 
> An easy solution would seem to be to have different perlbal instances
> handling different vips, but that introduces extra config files and what
> not.  It'd be nice to simply round-robin load balance between the two
> instances, without adding another full blown load balancer to the mix.
> 
> Any suggestions,warnings?
> 

I've done one large buildout using OpenBSD's PF to do stupid L4 load 
balancing between a couple perlbal machines. Each machine ran four 
instances.

I could recommend a few things:

Seriously look into having a stupid L4 load balancer in front. 
Eventually you'll want more than one perlbal machine anyway, and the 
stupid L4 machine can do firewalling/etc for you too. My own solution 
was to write a basic health check daemon which would modify PF tables 
live as perlbals went up and down. So traffic doesn't get redirected 
into a blackhole during rolling restarts and such.

You can either have the perlbals listen on different ports, or multiple 
IP's on the same box. I did multiple IP's because of how PF's round 
robin works.

It was easy to write a template for all of the perlbal configs and have 
a script apply the template and push the configs out. The template would 
paste in the IP address in a few spots and that was it.

You might also consider using the taskset utility to 'pin' each perlbal 
to a specific CPU. It's minor but it helps.

-Dormando