Perlbal with Mongrels

[TJ Murphy]

Using your patch, I did a little more digging.  Here's what I found:

Like you noted, Perlbal's verify_backend option defaults to doing an
HTTP OPTIONS request on the backend server at the path *

Mongrel doesn't like -any- request at the path *.  It throws an error
and doesn't respond.

When Perlbal doesn't get a response from the OPTIONS request, it goes
looking for another backend to use.  For my test setup I had only 1
backend, so it decided to reuse the same server.  It flooded Mongrel
with OPTIONS requests indefinitely.  It wasn't pretty.

Next, I set verify_backend_path to /.  Mongrel accepted this happily and
spit back a response as if it were an HTTP GET.

Unfortunately, I can't really afford to double the number of requests
I'm doing,  So I took it a step further.

I created a custom Mongrel handler insired by ErrCount, a fast cgi-style
counter in Rails (http://errtheblog.com/post/8444).  I set
verify_backend_path to /ctr and had the custom Mongrel handler listen on
/ctr.  All the handler does is pick up the request and spit out a blank
200 response (see attached).  It's fast.

To get this script working with a mongrel setup, just put the path to
the script in your mongrel_cluster.yml as config_script.   You can also
pass the script in on the command line using -S.

This setup has been running for over a week under really heavy load, and
it's holding up well.

-TJ



-----Original Message-----
From: Adam Jacob [mailto:adam at hjksolutions.com] 
Sent: Thursday, November 01, 2007 7:39 PM
To: TJ Murphy
Cc: perlbal at lists.danga.com
Subject: Re: Perlbal with Mongrels

I sent a patch to allow this to be a bit more tuneable, which allows
this to work with Rails.  (If anyone can figure out how to get Rails to
respond to an OPTION * request, you are my hero.)  It apparently got
lost in the shuffle for the last release, so here it is, updated for the
current trunk (should apply directly to 1.60 as well)

It allows you to set a verify_backend_path, which defaults to the
current '*'.  For rails, you can set it to /, or any other url you would
like to route your OPTIONs request to.

Perlbal + Mongrel with verify_backend solves a pretty irritating
behavior on mongrel's part, which is the caching of connections while it
waits on the Rails process to complete.  I'm using this patched-perlbal
in production on several different Rails apps, and it works great.

Regards,
Adam

On 10/31/07, TJ Murphy <TJ at freewebs.com> wrote:
>
>
> Allo,
>
> I'm currently running Perlbal (1.59, reverse_proxy) to load balance a 
> bunch of mongrels.  Occasionally I get to a point where Perlbal will 
> serve return a blank white screen instead of returning the app 
> response.  My users have dubbed it the "White Screen Of Death".  I'm 
> not sure where to begin debugging the problem or sorting out the
cause.
>
> I see the issue happen when I have a spike in requests or if I kill 
> off the mongrels.  I have found that adding more mongrels usually 
> leads to less white screens.
>
> Also, it seems that verify_backend doesn't really jive with mongrel.  
> I suspect it's because mongrel doesn't respond to an OPTIONS request.

> Is there any way to benefit from verify_backend in my case?


--
HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam
Jacob, Senior Partner
T: (206) 508-4759 E: adam at hjksolutions.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: options.rb
Type: application/octet-stream
Size: 260 bytes
Desc: options.rb
Url : http://lists.danga.com/pipermail/perlbal/attachments/20071105/420fed53/options.obj