Read & Append Files in MogileFS
dan at humyo.com
Mon Jun 25 22:22:51 UTC 2007
Thank you for your detailed response, I think I understand how to add this
now, please could you confirm that the following approach is correct and I
will get on with it and post the patch.
> -- you request via the protocol you'd like to edit "foo". the
> server then breaks off one of the 3 replicas of FID #100,
> forgetting about it for FID #100, allocates FID #101, and
> gives you that URL to work with for as long as you'd like
> (possible mogilefsd renames the file first, so the URL has
> 101.fid instead of 100.fid in the URL, but that doesn't
An edit_file method is added to MogileFS::Client which sends an edit_start
A cmd_edit_open() is added to MogileFS::Worker::Query which:
1. calls register_tempfile() for a new FID
2. gets a list of replicas using fid_devids()
3. removes the FID from one replica using remove_fidid_from_devid()
4. adds the new FID to the same replica using add_fidid_to_devid ()
5. returns the path to the replica with the new FID
Q: Is it nessessary to check any statuses of the replica in order to choose
which one to change the FID for?
MogileFS::Client renames the file to the new FID using WebDAV and returns a
> -- you do your DAV/otherwise edits to that URL,
Modify MogileFS::NewHTTPFile adding GET support, byte ranges in GET and PUT
> -- when you're done, you tell the server with create_close, as
> normal, as if you'd just written the whole file out, and
> the normal "'foo' now means file #101, not #100" logic
> kicks in, where #101 is replicated around, and #100 is
> cleaned up.
> I'm not sure what your application is, though, and if adding support for
> this wouldn't just tempt people to do things in a wrong/inefficient way.
> Can you explain more what you'd want to use this for?
For online file storage service http://www.humyo.com. Specifically:
1. Resumption of uploads - the receieved data must be appended to a file
already on the server.
2. Rsync patches to files - a user with low bandwidth may send an Rsync
delta rather than the whole file in which case we will need to write to the
file on the server in a variety of different locations in order to apply it.
More information about the mogilefs