how are files stored on disk?
eml at guba.com
Tue Jun 6 23:42:13 UTC 2006
Chuck Remes wrote:
> 1. How are files stored on disk and in what directory structure? Do
> they have meaningful names or do they use a unique hash as their name?
Their filename is their 'fid', which is the unique number that
represents the file in the database. They're placed in a directory
structure by breaking up the fid. It looks something like this:
> 2. Related to 1, if all files are stored in one (or very few)
> directory(ies), how does the node's file system performance scale? I'm
> interested because I realize many filesystems have trouble with
> directories containing thousands or tens of thousands of files. In
> practice is this a problem afflicting anyone?
I don't think you'll get more than a couple hundred files in any
particular dir. Don't recall the specifics, though.
We easily store hundreds of thousands of files without problems.
> 3. The list archives indicate there are problems with moving files
> larger than 50MB or so. I have an application where some files are
> several hundred MB or even in the low gigabytes. Is this a limitation
> of perlbal or some other mechanism? Would it be hard to swap the HTTP
> PUT mechanism with rsync or another tool better suited for large files?
I think Brad threw in my patch that prevents the timeout when 'PUT'ting
a really big file. With that in place, you shouldn't have any problems
until you go over 2GB, where you get a problem with the mogstored 'PUT'
handler using signed integers somewhere not returning correct filesizes
above the maximum size of a signed integer. (I haven't tried yet, but
I'm hoping the lighttpd patch that was just submitted might get around this)
More information about the mogilefs