Using MogileFS as a general replicated storage filesystem
brad at danga.com
Fri Mar 17 03:47:56 UTC 2006
On Thu, 16 Mar 2006, Tom Insam wrote:
> A combination of Filesys::Virtual and Net::DAV::Server will let you
> provide a virtual filesystem, backed by the Mogile store, and export it
yes, Filesys::Virtual and Net::DAV::Server are cool. Net::DAV::Server
needs some work, though. It could learn a thing or two from Apache's
mod_dav and the magic headers you send to Win/Mac to make them behave.
We (Danga) started working on this a year or so ago, so people could
mount their photos/files as if they were local (using Filesys::Virtual,
etc) but then we ran into the Net::DAV::Server immaturity and never
finished updating it. Did a lot of packet sniffing between Mac and .Mac
and WinXP and MSN Whatever service, though.
> as a DAV server (or an FTP server, using POE::Component::Server::FTP)
I can't imagine an event-based server (say POE) using an interface like
Filesys::Virtual that is synchronous will result in very good performance.
In fact, that's my biggest beef with POE people in general: they tend to
have very little respect for how terrible even minor blocking can cause
throughput in an event-based app to suffer. I'm not blaming you, because
I don't know you, but I'm just ranting, because today is rant day. :)
> Then you can mount it from some other machine (reasonably easy whatever
> your OS nowadays) and presto! it looks like a real filesystem. Plus, you
> can mount it from every machine on your network at once and treat it as
> shared storage.
Yes, very cool. Somebody should write this. :)
> Problems - this approach will turn out to be almost exclusively nasty
> edge cases. Every DAV client I've seen recently is horrible in a new and
> unpleasant way, and Filesys::Virtual isn't very mature, and doesn't make
> this easy. It may also be very slow compared to a real filesystem, again
> mostly because OS-integrated DAV clients (I'm talking to you, MacOS X
> Finder) treat the mount like a normal disk, stat everything in sight,
> create metadata files, and thumbnail everything that holds still, and
> this hurts the mount server. But it'll work, and be Free.
Immature, yes, but I wouldn't worry about "speed". Disks are the
bottleneck for the past many years, not CPU.
> Hmm. Having said that about the single mount point... DAV is of course
> HTTP. It's stateless. And we're quite good at load-balancing stateless
> HTTP-based protocols....
More information about the mogilefs