Input on best practices for serving files

J. Shirley jshirley at gmail.com
Wed Apr 9 22:17:26 UTC 2008


Hey all,

I've been playing with MogileFS for quite some time in testing, but
I'm getting ready to actually do something with it now and have been
searching for some "best practices" regarding serving the stored
mogileFS files.  I've come up empty searching for this information
aside from some mailing list posts, so I figure that I just need to
get some input, write up my assumptions and then I'd like to post to
the wiki.  If this is good enough, who do I ask for an account to
update the wiki (seems I need an invite key)?  After it is up there, I
hope people can just make incremental revisions to get it into better
shape.

To begin with, it is recommended to write an "interpretation" layer
that translates the end-user URL to an internal mogileFS layer.  For
this example, a url like
"http://static.myapp.com/users/1234567-thumb.jpg" would then be
translated to a mogileFS stored key, along with the class.  To keep
this simple, the key could be "1234567:thumb" and the class would be
"users".

The next step is to write some code that takes the key and class,
fetches the paths from a tracker and then, in some fashion, proxies
the image response.

There are two distinct points where caching can be utilized to speed
up the serving of files.  Up front, you can cache the entire URL (so
that if http://static.myapp.com/users/1234567-thumb.jpg has been seen,
it returns the last response, just like any other image proxy would
work).  Secondarily, cache the resulting URLs returned from the
tracker.  This is where memcached plays in nicely together with
mogileFS.

I'd also like to provide some sample code for fetching the paths and
reproxying, but I'm uncertain of the best way to do this and I'm
hoping others can help.  If you just forward the request to the
internal tracker URL, I don't quite see how to set the mime-type and
other headers (perhaps this is something easy in perlbal?  I have to
admit my ignorance on this.)

The other point that I'd like to talk about is an image manipulation
layer.  Something like how most image hosting services do that checks
the incoming referrer header.  If the referrer is blank, or the host
is not equal to "myapp.com", then affix a "stamp" on to the top of the
image.  Has there been any thoughts for a mogileFS Cookbook setup?  I
think it would greatly help out newcomers to the product.

Thanks,
-Jay


More information about the mogilefs mailing list