You can md5 hash the content and use that as the key? That's what we do. Our files are typically 50-100MB in size so having just 1 copy of the file on the cluster makes saves space (some content is shared between users). Only time we loose is if the file needs to be in multiple domains, but then you might not want to share anything between domains anyway. Also, the overhead of calculating the MD5 is not a problem for us but might be unacceptable for you.
<br><br>Most of what you want can be done with just a few lines of code in application space.. the I/O win would be pretty minimal, unless the majority of your opertations is copying files, in which case MogileFS might not even be the right solution anyway ;)
<br><br>Gr,<br>Andy<br><br><div class="gmail_quote">On Nov 9, 2007 3:43 PM, John Berthels <<a href="mailto:jjberthels@gmail.com">jjberthels@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="Ih2E3d">On 09/11/2007, Mark Smith <<a href="mailto:smitty@gmail.com">smitty@gmail.com</a>> wrote:<br>> > I'd like to add a copy_file cmd to mogilefs (assuming there isn't one<br>> > already?).
<br>><br>> What's the use case you're going for here / what problem are you<br>> trying to solve?<br><br></div>We maintain a level of indirection between user-level files and<br>back-end stored files, so it is possible for two user-level files to
<br>actually refer to the same back-end storage.<br><br>This allows us to save a few % in shared storage (copied files share<br>storage, we also merge files with identical contents). We need to<br>break the copy on the (fairly rare) case of a update to a file (we
<br>make use of the edit_file capability). Copy-on-write, effectively.<br><br>In that case, we need to create a new copy of the file contents. We<br>can do this with an app-level read/write loop, but mogile is in a<br>position to do this with less overall I/O (and potentially much lower
<br>latency, since the common case requires no bulk I/O before returning<br>to the user - just breaking off a replica and giving it a new fid).<br><br>It could also provide a way (other than a read/write loop) to move a<br>
file from one domain/class to another (copy then delete).<br><font color="#888888"><br>jb<br></font></blockquote></div><br>