Using MogileFS as a general replicated storage filesystem

[Brad Fitzpatrick]

On Thu, 16 Mar 2006, Tom Insam wrote:

> A combination of Filesys::Virtual and Net::DAV::Server will let you
> provide a virtual filesystem, backed by the Mogile store, and export it

yes, Filesys::Virtual and Net::DAV::Server are cool.  Net::DAV::Server
needs some work, though.  It could learn a thing or two from Apache's
mod_dav and the magic headers you send to Win/Mac to make them behave.
We (Danga) started working on this a year or so ago, so people could
mount their photos/files as if they were local (using Filesys::Virtual,
etc) but then we ran into the Net::DAV::Server immaturity and never
finished updating it.  Did a lot of packet sniffing between Mac and .Mac
and WinXP and MSN Whatever service, though.

> as a DAV server (or an FTP server, using POE::Component::Server::FTP)

I can't imagine an event-based server (say POE) using an interface like
Filesys::Virtual that is synchronous will result in very good performance.
In fact, that's my biggest beef with POE people in general:  they tend to
have very little respect for how terrible even minor blocking can cause
throughput in an event-based app to suffer.  I'm not blaming you, because
I don't know you, but I'm just ranting, because today is rant day.  :)

> Then you can mount it from some other machine (reasonably easy whatever
> your OS nowadays) and presto! it looks like a real filesystem. Plus, you
> can mount it from every machine on your network at once and treat it as
> shared storage.

Yes, very cool.  Somebody should write this.  :)

> Problems - this approach will turn out to be almost exclusively nasty
> edge cases. Every DAV client I've seen recently is horrible in a new and
> unpleasant way, and Filesys::Virtual isn't very mature, and doesn't make
> this easy. It may also be very slow compared to a real filesystem, again
> mostly because OS-integrated DAV clients (I'm talking to you, MacOS X
> Finder) treat the mount like a normal disk, stat everything in sight,
> create metadata files, and thumbnail everything that holds still, and
> this hurts the mount server. But it'll work, and be Free.

Immature, yes, but I wouldn't worry about "speed".  Disks are the
bottleneck for the past many years, not CPU.

> Hmm. Having said that about the single mount point... DAV is of course
> HTTP. It's stateless. And we're quite good at load-balancing stateless
> HTTP-based protocols....

Indeed!

- Brad