perlbal use case question

[Nathan Kurz]

On Tue, Jun 10, 2008 at 8:34 AM, Michael Engelhart
<mike.engelhart at gmail.com> wrote:
> I'm looking into Perlbal to solve a particular problem with a new web
> services platform that we're developing.

I'm new to Perlbal, but I'll take a stab as I've been looking at using
it for something similar (aggregating search results from multiple
shards on a local network).

First, let me see if I understand correctly what you are trying to do:
1) You get a user request, which maps to multiple external requests.
2) You make these slow requests and wait for the responses.
3) You aggregate these results and return them to the user.

And you are planning to do this by:
1) Perlbal proxies the request to Apache which proxies to your application
2) Application sends back Perlbal redirect with payload of URLs.
3) Perlbal gives URLs to one of a pool of fetchers (or is it one per URL?).
4) Fetcher returns Perlbal redirect with aggregated XML responses as payload.
5) Perlbal redirects back to Apache which proxies to application to finish.

If this is what you are doing, my instinct is that Perlbal's internal
redirects would work for you, but your architecture has more moving
parts than you really need.  For example, I'm not sure what role
Apache is playing for you here, and I'd guess that a very small amount
of code would allow Perlbal to make and wait for the outside requests
in a nice non-blocking way.  This might just be prejudice on my part,
as I'd be more comfortable extending Perlbal a small amount rather
than writing that custom lightweight process.

My model would probably look more like this:
1) Perlbal proxies user request directly to one of a pool of app servers.
2) App server does some work and returns a 'custom' redirect and a list of URLs.
3) Custom Perlbal plugin queues and makes asynchronous external requests.
4) Collected responses are redirected to a second pool of app servers.
5) Second level app server homogenizes responses and generated finished XML.

The first and second level app servers could be the same pool of
processes with a different request structure, but barring odd
circumstances it strikes me as easier and more scalable to keep them
separate.

Or perhaps I misunderstand your goals.    It probably could also be
done using nothing but Perlbal redirects as you describe.

Hope this helps,

Nathan Kurz
nate at verse.com