Possible use for memcache?

[David Phillips]

On 11/27/05, mike <mike503 at gmail.com> wrote:
> I'm just brain dumping here. Note that I am not that advanced of a
> programmer (I'm a PHP scripter) so filesystem and hardware I/O
> semantics are not my specialty; this is just what I've got from
> Googling until I'm blue in the face and why multi-path storage
> solutions are still not very cheap/open source (only a couple
> possibilities, each with limitations and caveats...)

First off, and generally unrelated to the subject at hand: memcached
is a cache.  Period.  If you ever think of doing anything with it
besides caching data, stop.  More than likely it is a very bad idea.

What you are asking for is the "holy grail" of distributed file
systems.  The reason that nothing you want exists for free is that the
problem is very difficult.  Every approach is going to have trade
offs.  Distributed computing is all about identifying and minimizing
trade offs.

Depending on what you want, exactly, which was not clear in your
email, you should look into AFS, Coda and Lustre.  Red Hat GFS and
Oracle Cluster FS might also be worth looking into.

One approach similar to MogileFS is the Google File System.  Read the
paper, then understand the design and the trade offs.  You will be
amazed at the complexity and it still isn't exactly what you want.

If you want to connect machines to a cluster, have them add their
local storage to the pool, mount the file system and have it all
magically work, then you will be disappointed.  Nothing like that
exists and I'm not sure it can.

An interesting approach might be Solaris' ZFS with network block
devices, maybe using ATA over Ethernet (AoE).  I wonder if anyone has
tried that?

--
David Phillips <david at acz.org>
http://david.acz.org/