Possible use for memcache?
mike503 at gmail.com
Sun Nov 27 18:48:55 PST 2005
I've been looking for clustered FS or (if I have to) clustered NFS
solutions, and there's not many out there.
However, I think I've figured out a way that might work; basically
using the memcache idea of hashing the requests to identify which
server is serving the data could alleviate any issues with NFS not
being "globally" aware, and allow for multiple NFS servers to read
from the same physical shared storage (note that the expectation is
the shared storage is being accessed from multiple physical servers,
and redundancy is done below the actual FS layer)
If someone could write an NFS client using FUSE (perhaps?) and have it
integrate with memcache, couldn't that basically allow for n+1 scaling
out of NFS servers and storage? Basically any time the FS would need
to issue or check a file lock, it would hit the memcache API to check
if it's been locked or "represented" yet by a specific NFS server.
I'm just brain dumping here. Note that I am not that advanced of a
programmer (I'm a PHP scripter) so filesystem and hardware I/O
semantics are not my specialty; this is just what I've got from
Googling until I'm blue in the face and why multi-path storage
solutions are still not very cheap/open source (only a couple
possibilities, each with limitations and caveats...)
Perhaps someone could shed some light, tell me it's stupid, or tell me
it's worthwhile? I know this is the memcache list, and not a
filesystems list, but perhaps it might spur some discussion.
Also, I know Danga has MogileFS, which sort of does this in a way, but
IMHO using HTTP over TCP is additional overhead I'd rather not bother
with, and it's still application-specific. Apps must use the MogileFS
library or it's useless (unless of course a FUSE layer was created...)
- NFS is pretty much tried and true, adding a small middle layer in
between to allow for distribution of NFS requests and alleviating the
lock conflicts might be a much more usable solution?
More information about the memcached