serving configuration

[Mark Smith]

> I know this has been touched on before, but I'm getting hung up on serving
> files (specifically step 4 in
> http://lists.danga.com/pipermail/mogilefs/2008-April/001571.html)
>
>  Right now, I'm able to write my files to mogileFS through my app using the
> python client.
>
> We have a bunch of apache servers load-balanced behind perlbal running on a
> separate machine. We have another box running lighttpd serving static files.
>
> The thing that's tripping me up is the whole user-facing URL -> mogile path
> -> actual file data step. It seems really overkill to me to run a separate
> perlbal instance, another instance of my django app (or a stripped down
> version of it) to construct the correct headers and get the URL, just to
> serve an image file (not to mention figuring out where to plug in memcached
> so I'm not bouncing requests all over my architecture).

Why would you run a second path?

Typically, the code that serves your application pages will also do
the translation of the URLs.  You just configure your Django app so
that if a request comes in for /some/known/path/* that corresponds to
your images, you do MogileFS translation.  You could do it by path or
domain.  Let's say that you decide:

Application path: http://myfoo.com/
Image path: http://myfoo.com/img/

if ( $uri begins with http://myfoo.com/img/ ) then
    # translate $uri to a MogileFS URI using MogileFS tracker
    # construct new headers for this image you're going to serve
        # Content-type is the big one you want to decide
    # append header X-REPROXY-URL: $translated_paths
    # return this request
else
    # dispatch to standard application path
end if

If this doesn't make sense, or is confusing, please let me know and I
can write up some pseudocode to try to elaborate on how this works.

> I'm sure that there
> are ways to configure mogile so that it's as fast as just serving an image
> file from lighttpd, right?

A user talking directly to lighttpd and fetching a local file is just
about the fastest path I can imagine outside of direct caching on your
load balancer (assuming you have one).

MogileFS is no turtle, but it's designed to solve other problems - not
really speed.  Flexibility, reliability, generally horizontal
expansion, that is what MogileFS does well.  If you are storing
millions of files for your users, it's probably a great fit.  If
you've got thousands of files of a few gigabytes, local storage and
serving from lighttpd is going to be faster.

> Are there any pre-built, lightweight tools for this step?

nginx as someone mentioned has some of the functionality you're
looking at.  Doesn't seem to be a good fit though since the rest of
your infrastructure is built around Perlbal/Django, I don't think you
need to go that path.  (But explore it, definitely!  More knowledge
never hurts, and from what I've heard/seen, nginx is very capable!)


-- 
Mark Smith / xb95
smitty at gmail.com