Questions about MogileFS in a production environment

Oleg Tyaglo otyaglo at
Tue Dec 25 23:17:10 UTC 2007

We have quite large mogile storage setup running in production.
Our current mogile cluster has 44T of storage where 19T used.
Total number of stored files is ~330 millions (x3 copies ~= 1 billion

We customized mogile cluster next way:
- 8 federated db boxes with backup mysql replication
- 2 trackers per db group (16 total)
- 23 storage nodes, each has 8 250-320G drives (one for each group)

Our performance is 
- 50 writes per second per whole cluster (tested up to 500 wps) 
- 120+ reads per second per storage node (~ 3000 reads per second on a
- 80% full device recovery (re-replication) usually takes 30 hours with
no affect to production performance.

We developed custom web gui tool and init.d scripts to manage multiple
(8) instances of storage daemons (mogstored) on a given storage node.
GUI tool also comes handy with graphical display of usage pattern and
device/storage node availability. 

In addition, we designed custom nagios checks that trigger notifications
about tracker/storage node availability along with single drive (device)

I hope you find this helpful.


-----Original Message-----
From: mogilefs-bounces at
[mailto:mogilefs-bounces at] On Behalf Of Justin Huff
Sent: Thursday, December 13, 2007 9:42 AM
To: Clinton Goudie-Nice
Cc: mogilefs
Subject: Re: Questions about MogileFS in a production environment

I'm not sure we'd count as 'large scale' -- but we use it at Picnik.

We have about 5.7TB, using about 2.7TB.  That's about 6.6million files
with a replcount of 2.


Clinton Goudie-Nice wrote:
> Dormando was kind enough to point me at some info in the mailing list
> from a few weeks ago where people talked about the scale of the
> stores they're currently using.
> Could anyone mention some names of some reasonably large scale web
> sites/services which I might recognize that are using MogileFS?
> Also, I've read a bit about trouble serving files greater than 100mb.
> there some limit on file size that I should be aware of?
> Thanks again!
> Clint
> Twelve Horses <> 	Mobile
> <> 	Social
> <>
> <> 	Email
> <>
> *Clinton Goudie-Nice*
> Architect / Senior Software Engineer
> cgoudie at <mailto:cgoudie at>
> Phone: 	+1.801.571.2665 ext 3264
> Mobile: 	+1.801.915.0629
> Fax: 	+1.801.571.2669
> LinkedIn:
> *Twelve Horses*
> 13961 Minuteman Drive
> Suite 125
> Draper, UT 84020
> * <>*
> On Wed, 2007-12-12 at 16:06 -0700, Clinton Goudie-Nice wrote:
>> Is anyone on the list using mogile in a large scale production
>> environment?
>> I'm considering running MogileFS in a production environment; sort of
>> a giant disk in the sky, for
>> an assortment of Internet apps. It will potentially be serving up
>> terabytes of data, mostly in small image files, probably behind a
>> squid caching proxy.
>> Is there anything people on the list could comment on? How stable is
>> it, how big are your various production stores, etc...
>> Thanks!
>> Clint

More information about the mogilefs mailing list