Multithreaded status (Was: Re: best way to upgrade)

[Steven Grimm]

Don MacAskill wrote:
> Sorry to thread hi-jack, but I'd love to hear the status of the 
> multithreaded version and whether it's in heavy production use at 
> Facebook.  And, if so, how it's doing, of course.  :)

We have been running it exclusively on all our production servers for 
several months (though without the changes that have landed in that 
branch in the last couple weeks). We found one minor bug in late 
February, but aside from that it has been trouble-free for us.

It gets extremely heavy usage; during peak times some instances handle 
over 60,000 requests per second sustained, about 95% of which are "get" 
requests. We haven't hit a bottleneck yet, but at the moment it appears 
that the limiting factor will be the Linux kernel's interrupt handling. 
It appears to handle all the interrupts from incoming packets on one 
CPU, as we can see the system CPU time on one of the four CPUs exceed 
50% while the other three are more in the 20% range. If anyone knows how 
to spread that load around, I'd be interested in that (admittedly I 
haven't actually gone out and tried to research it yet; for all I know 
it's simple to do.)

-Steve