Questions about MogileFS in a production environment

[Oleg Tyaglo]

We have quite large mogile storage setup running in production.
Our current mogile cluster has 44T of storage where 19T used.
Total number of stored files is ~330 millions (x3 copies ~= 1 billion
objects)

We customized mogile cluster next way:
- 8 federated db boxes with backup mysql replication
- 2 trackers per db group (16 total)
- 23 storage nodes, each has 8 250-320G drives (one for each group)

Our performance is 
- 50 writes per second per whole cluster (tested up to 500 wps) 
- 120+ reads per second per storage node (~ 3000 reads per second on a
whole
cluster)
- 80% full device recovery (re-replication) usually takes 30 hours with
no affect to production performance.

We developed custom web gui tool and init.d scripts to manage multiple
(8) instances of storage daemons (mogstored) on a given storage node.
GUI tool also comes handy with graphical display of usage pattern and
device/storage node availability. 

In addition, we designed custom nagios checks that trigger notifications
about tracker/storage node availability along with single drive (device)
failure.

I hope you find this helpful.

Oleg


-----Original Message-----
From: mogilefs-bounces at lists.danga.com
[mailto:mogilefs-bounces at lists.danga.com] On Behalf Of Justin Huff
Sent: Thursday, December 13, 2007 9:42 AM
To: Clinton Goudie-Nice
Cc: mogilefs
Subject: Re: Questions about MogileFS in a production environment

I'm not sure we'd count as 'large scale' -- but we use it at Picnik.

We have about 5.7TB, using about 2.7TB.  That's about 6.6million files
with a replcount of 2.

--Justin

Clinton Goudie-Nice wrote:
> Dormando was kind enough to point me at some info in the mailing list
> from a few weeks ago where people talked about the scale of the
MogileFS
> stores they're currently using.
> 
> Could anyone mention some names of some reasonably large scale web
> sites/services which I might recognize that are using MogileFS?
> 
> Also, I've read a bit about trouble serving files greater than 100mb.
Is
> there some limit on file size that I should be aware of?
> 
> Thanks again!
> 
> Clint
> 
> 
> Twelve Horses <http://web.twelvehorses.com/> 	Mobile
> <http://web.twelvehorses.com/info_center/whitepaper/> 	Social
> <http://web.twelvehorses.com/solutions/blogging_for_business/>
Web
> <http://web.twelvehorses.com/solutions/website_design/> 	Email
> <http://web.twelvehorses.com/technology/>
> 
> 
> *Clinton Goudie-Nice*
> Architect / Senior Software Engineer
> cgoudie at twelvehorses.com <mailto:cgoudie at twelvehorses.com>
> Phone: 	+1.801.571.2665 ext 3264
> Mobile: 	+1.801.915.0629
> Fax: 	+1.801.571.2669
> LinkedIn: 	http://www.linkedin.com/in/cgoudie
> 
> 	
> *Twelve Horses*
> 13961 Minuteman Drive
> Suite 125
> Draper, UT 84020
> *www.twelvehorses.com <http://www.twelvehorses.com>*
> 
> 
> 
> 
> 
> On Wed, 2007-12-12 at 16:06 -0700, Clinton Goudie-Nice wrote:
>> Is anyone on the list using mogile in a large scale production
>> environment?
>>
>> I'm considering running MogileFS in a production environment; sort of
>> a giant disk in the sky, for
>> an assortment of Internet apps. It will potentially be serving up
>> terabytes of data, mostly in small image files, probably behind a
>> squid caching proxy.
>>
>> Is there anything people on the list could comment on? How stable is
>> it, how big are your various production stores, etc...
>>
>> Thanks!
>>
>> Clint
>