Using MogileFS as a general replicated storage filesystem

Tom Insam tom at jerakeen.org
Thu Mar 16 22:50:22 UTC 2006


On Thu, 2006-03-16 at 22:25 +0100, Robin Smidsrød wrote:

> So you can understand that I got very excited when I noticed MogileFS in 
> a comment on archive.org's pages about the PetaBox. It looked like 
> someone had finally implemented something similar to the GoogleFS in 
> Open Source. I got very curious indeed. After looking over the website 
> for MogileFS I noticed that it wasn't a traditional filesystem, but it 
> had a lot of similarities. I noticed that it was implemented in Perl (my 
> language, perfect!) and I got even more curious. But then I got sad 
> again when I noticed that you tried implementing a wrapper with FUSE, 
> and you kinda gave up on it.

ooooh, ok, evil thoughts here. Although MogileFS isn't a traditional
filesystem, it will store files across random drives, replicate them,
etc. But you know that.

Let's assume for the sake of argument that you don't need
high-availability on your filesystem - Mogile will provide reliable file
storage, but my plan involves a single machine that pretends to hold the
filesystem proper, and if it goes down, you'll lose the filesystem till
you prop it up again.

A combination of Filesys::Virtual and Net::DAV::Server will let you
provide a virtual filesystem, backed by the Mogile store, and export it
as a DAV server (or an FTP server, using POE::Component::Server::FTP).
Then you can mount it from some other machine (reasonably easy whatever
your OS nowadays) and presto! it looks like a real filesystem. Plus, you
can mount it from every machine on your network at once and treat it as
shared storage.

Problems - this approach will turn out to be almost exclusively nasty
edge cases. Every DAV client I've seen recently is horrible in a new and
unpleasant way, and Filesys::Virtual isn't very mature, and doesn't make
this easy. It may also be very slow compared to a real filesystem, again
mostly because OS-integrated DAV clients (I'm talking to you, MacOS X
Finder) treat the mount like a normal disk, stat everything in sight,
create metadata files, and thumbnail everything that holds still, and
this hurts the mount server. But it'll work, and be Free.

Hmm. Having said that about the single mount point... DAV is of course
HTTP. It's stateless. And we're quite good at load-balancing stateless
HTTP-based protocols....

tom




More information about the mogilefs mailing list