copy cmd proposal

[Andy Lo A Foe]

You can md5 hash the content and use that as the key? That's what we do. Our
files are typically 50-100MB in size so having just 1 copy of the file on
the cluster makes saves space (some content is shared between users). Only
time we loose is if the file needs to be in multiple domains, but then you
might not want to share anything between domains anyway. Also, the overhead
of calculating the MD5 is not a problem for us but might be unacceptable for
you.

Most of what you want can be done with just a few lines of code in
application space.. the I/O win would be pretty minimal, unless the majority
of your opertations is copying files, in which case MogileFS might not even
be the right solution anyway ;)

Gr,
Andy

On Nov 9, 2007 3:43 PM, John Berthels <jjberthels at gmail.com> wrote:

> On 09/11/2007, Mark Smith <smitty at gmail.com> wrote:
> > > I'd like to add a copy_file cmd to mogilefs (assuming there isn't one
> > > already?).
> >
> > What's the use case you're going for here / what problem are you
> > trying to solve?
>
> We maintain a level of indirection between user-level files and
> back-end stored files, so it is possible for two user-level files to
> actually refer to the same back-end storage.
>
> This allows us to save a few % in shared storage (copied files share
> storage, we also merge files with identical contents). We need to
> break the copy on the (fairly rare) case of a update to a file (we
> make use of the edit_file capability). Copy-on-write, effectively.
>
> In that case, we need to create a new copy of the file contents. We
> can do this with an app-level read/write loop, but mogile is in a
> position to do this with less overall I/O (and potentially much lower
> latency, since the common case requires no bulk I/O before returning
> to the user - just breaking off a replica and giving it a new fid).
>
> It could also provide a way (other than a read/write loop) to move a
> file from one domain/class to another (copy then delete).
>
> jb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/mogilefs/attachments/20071109/a3b333d0/attachment.html