Running multiple perlbal instances

Ryan Woodrum rwoodrum at slard.net
Wed Jan 30 00:48:21 UTC 2008


On Tuesday 29 January 2008 03:27:07 pm Ask Bjørn Hansen wrote:
> On Jan 29, 2008, at 12:49, Kevin Olson wrote:
> > Does anyone have some advice on running multiple perlbal instances
> > on one
> > multi-cpu box?  We're getting to the point where one perlbal
> > instance is
> > handling 30,000-45,000 connections/min and load is hitting 1.00.
>
> (I'm assuming you already use the XS headers module; otherwise that's
> an easy win).
>
> I haven't tried it, but doesn't the Linux iptables have a "random"
> module?  If so then you should be able to redirect half the new
> connections to one port and the other half to the other.
>
> If you use FreeBSD or OpenBSD then I imagine pf has a "random" thing,
> too.
>
>
>   - ask

I think you could do something like this with iptables pretty easily.  If you 
run perlbal on multiple ip's/interfaces, you could set up different routing 
tables and mark packets matching a criteria and then sending them to a 
certain routing table.  For example, if you want two instances of perlbal 
running, you could match on the last bit in the last octet of the source ip 
address.  I have implemented this sort of poor-man's load balancing on 
clusters of content filters that process all traffic from all k-12 schools in 
FL (boatloads of traffic).  It works very well, is exceptionally fast, and is 
relatively easy to configure.

You would match with an iptables rule like:
-A PREROUTING -s 0.0.0.0/0.0.0.1 --set-mark 0x02
-A PREROUTING -s 0.0.0.1/0.0.0.1 --set-mark 0x03

You would send marked packets to certain routing tables:
ip rule add fwmark 0x2 table 2
ip rule add fwmark 0x3 table 3

You would add routes for these tables to your perlbal instances:
ip route add default via 172.16.1.100 table 0x2
ip route add default via 172.16.1.101 table 0x3


Alternatively if you didn't want to use multipe ip's, you could use iptables 
to dnat on an interface with multiple "--to-destination"'s specified.  When 
multiple "--to-destinations" are specified, the kernel will simply 
round-robin between them.  In this case, you would listen for incoming 
traffic and simply send it to your various perlbal instances.  This is 
actually mangling the packet, however, which may not be ideal or suitable.

e.g. if your public address is 4.4.4.4 and you have perlbal vips running on 
172.16.1.100:8080 and 172.16.1.100.8081:
-t nat -A PREROUTING -i eth0 -d 4.4.4.4 --dport 80 -j DNAT --to-destination \    	
172.16.1.100:8080 --to-destination 172.16.1.100:8081 ...

In either of these cases, a common gotcha with using lots of connection 
tracking inside of iptables is to properly configure the module with enough 
buckets and to perhaps reduce the automatic timeout value.

Anyway... this is primarily me rattling off the top of my head from a previous 
implementation, so it's certainly not a step-by-step guide.  I'd be more than 
happy to give some more detailed info if desired.

-ryan woodrum


More information about the perlbal mailing list