Running multiple perlbal instances
rwoodrum at slard.net
Wed Jan 30 00:48:21 UTC 2008
On Tuesday 29 January 2008 03:27:07 pm Ask Bjørn Hansen wrote:
> On Jan 29, 2008, at 12:49, Kevin Olson wrote:
> > Does anyone have some advice on running multiple perlbal instances
> > on one
> > multi-cpu box? We're getting to the point where one perlbal
> > instance is
> > handling 30,000-45,000 connections/min and load is hitting 1.00.
> (I'm assuming you already use the XS headers module; otherwise that's
> an easy win).
> I haven't tried it, but doesn't the Linux iptables have a "random"
> module? If so then you should be able to redirect half the new
> connections to one port and the other half to the other.
> If you use FreeBSD or OpenBSD then I imagine pf has a "random" thing,
> - ask
I think you could do something like this with iptables pretty easily. If you
run perlbal on multiple ip's/interfaces, you could set up different routing
tables and mark packets matching a criteria and then sending them to a
certain routing table. For example, if you want two instances of perlbal
running, you could match on the last bit in the last octet of the source ip
address. I have implemented this sort of poor-man's load balancing on
clusters of content filters that process all traffic from all k-12 schools in
FL (boatloads of traffic). It works very well, is exceptionally fast, and is
relatively easy to configure.
You would match with an iptables rule like:
-A PREROUTING -s 0.0.0.0/0.0.0.1 --set-mark 0x02
-A PREROUTING -s 0.0.0.1/0.0.0.1 --set-mark 0x03
You would send marked packets to certain routing tables:
ip rule add fwmark 0x2 table 2
ip rule add fwmark 0x3 table 3
You would add routes for these tables to your perlbal instances:
ip route add default via 172.16.1.100 table 0x2
ip route add default via 172.16.1.101 table 0x3
Alternatively if you didn't want to use multipe ip's, you could use iptables
to dnat on an interface with multiple "--to-destination"'s specified. When
multiple "--to-destinations" are specified, the kernel will simply
round-robin between them. In this case, you would listen for incoming
traffic and simply send it to your various perlbal instances. This is
actually mangling the packet, however, which may not be ideal or suitable.
e.g. if your public address is 220.127.116.11 and you have perlbal vips running on
172.16.1.100:8080 and 172.16.1.100.8081:
-t nat -A PREROUTING -i eth0 -d 18.104.22.168 --dport 80 -j DNAT --to-destination \
172.16.1.100:8080 --to-destination 172.16.1.100:8081 ...
In either of these cases, a common gotcha with using lots of connection
tracking inside of iptables is to properly configure the module with enough
buckets and to perhaps reduce the automatic timeout value.
Anyway... this is primarily me rattling off the top of my head from a previous
implementation, so it's certainly not a step-by-step guide. I'd be more than
happy to give some more detailed info if desired.
More information about the perlbal