Data splitting

[Pascal Hakim]

On Tue, 2007-08-07 at 11:57 +0800, drpr0ctologist at gmail.com wrote:
>         You need a webserver (or a fancy Perlbal plugin) ... Perlbal
>         and MogileFS
>         alone won't do this.
>         
>         I repeat:  Perlbal does not talk to mogilefsd.  (only to
>         mogstored, when
>         you tell it to, from your web server)
> 
> On the Perlbal page, I see that Perlbal can act as a web server? Or is
> it only via some plugin?

It can act as a web server, but the usage being discussed here is using
perlbal as a reverse proxy/load balancer

> I think I'm asking way too many questions here, but that's because I
> can't seem to find any documentation or detailed tutorials on how to
> set Perlbal up and have it work with Apache + MogileFS.  Is there such
> a tutorial?
> 

The basic idea is:

When you set perlbal up, make the hosts it's listing to have a role of
reverse_proxy. You can view config options are at:
http://search.cpan.org/src/BRADFITZ/Perlbal-1.59/doc/service-parameters.txt

Also look at "Role: Reverse Proxy" on http://www.danga.com/perlbal/


Setup perlbal to have reproxy support:
http://search.cpan.org/src/BRADFITZ/Perlbal-1.59/doc/reproxying.txt

When you get a request, you pass it on to your backend node (which can
be running anything you want, ie Apache or whatever).

The backend node that it talks to will talk to mogilefsd, and get the
location of the file. You can then do any logic you need in there (ie,
access control and the like). Your code on the backend node can then
return X-REPROXY-URL with the mogstored paths that your backend node got
out of mogilefsd.

Your backend node then moves on to something else, and perlbal/mogstored
take care of spitting the file out back to the client.

By doing this setup, you gain horizontal scalability (You can add more
and more nodes, as you need more capacity), as well as a better ability
to deal with failure if one of your mogstored server fails (you have all
those files on other nodes). 

Cheers,

Pasc