more then 4 billion files?

[Nathan Schmidt]

Blake,

 From a practical perspective you'll probably need to split your  
MogileFS database instances (and hence trackers) to keep the indices  
(to say nothing of the file and file_on tables) a manageable size.  
Keep in mind your file_on table will have {mindevicecount}x more rows  
than you have files.

Here's some real-world data:

 > show table status;
...
for 'file':
	Data_length = 1,026,017,724
	Index_length = 1,170,308,096

for 'file_on':
	Data_length = 228,408,088
	Index_length = 792,318,976


Not really huge datasets but this is for only 14M files at  
mindevicecount=2. Tonight I'm adding millions more and bumping to   
mindevicecount=3 and I expect that footprint will rise accordingly.

In your case 4B = 28x more file entries so that's ~30Gb for just the  
'file' index -- keeping that whole index in core probably means  
either Really Big Servers or addressing multiple pools of MogileFS.

I'd hate to hit a hard wall with one 4B-entry table and _then_ decide  
it's time to fix the client side.

Regards,
-Nathan / PBwiki

On Feb 23, 2007, at 1:19 AM, Blake Golliher wrote:

> Any one using Mogile on more then 4 billion files?  If not, anyone
> think there's any design limitation on the maximum number of objects
> MogileFS will support?  Or is it just something someone has to test?
> Specifically in the user uploaded content space, written once, read
> for ever, and ever and ever.
>
> -Blake