how are files stored on disk?

Eric Lambrecht eml at guba.com
Tue Jun 6 23:42:13 UTC 2006


Chuck Remes wrote:
> 1. How are files stored on disk and in what directory structure? Do  
> they have meaningful names or do they use a unique hash as their name?

Their filename is their 'fid', which is the unique number that 
represents the file in the database. They're placed in a directory 
structure by breaking up the fid. It looks something like this:

	00/12/34/001234.fid

> 2. Related to 1, if all files are stored in one (or very few)  
> directory(ies), how does the node's file system performance scale?  I'm 
> interested because I realize many filesystems have trouble with  
> directories containing thousands or tens of thousands of files. In  
> practice is this a problem afflicting anyone?

I don't think you'll get more than a couple hundred files in any 
particular dir. Don't recall the specifics, though.

We easily store hundreds of thousands of files without problems.

> 3. The list archives indicate there are problems with moving files  
> larger than 50MB or so. I have an application where some files are  
> several hundred MB or even in the low gigabytes. Is this a limitation  
> of perlbal or some other mechanism? Would it be hard to swap the HTTP  
> PUT mechanism with rsync or another tool better suited for large files?

I think Brad threw in my patch that prevents the timeout when 'PUT'ting 
a really big file. With that in place, you shouldn't have any problems 
until you go over 2GB, where you get a problem with the mogstored 'PUT' 
handler using signed integers somewhere not returning correct filesizes 
above the maximum size of a signed integer. (I haven't tried yet, but 
I'm hoping the lighttpd patch that was just submitted might get around this)

Eric...


More information about the mogilefs mailing list