Questions about MogileFS in a production environment

[Andre' Hazelwood]

Oleg,
What kind of hardware makes up each machine if you don't mind my asking?
Also, any other advice you can give would be most appreciated.  That
would be helpful for anyone like us that is looking at the best way to
implement mogilefs.

Warmest regards,
Andre'

-----Original Message-----
From: mogilefs-bounces at lists.danga.com
[mailto:mogilefs-bounces at lists.danga.com] On Behalf Of Oleg Tyaglo
Sent: Tuesday, December 25, 2007 6:17 PM
To: Justin Huff; Clinton Goudie-Nice
Cc: mogilefs
Subject: RE: Questions about MogileFS in a production environment

We have quite large mogile storage setup running in production.
Our current mogile cluster has 44T of storage where 19T used.
Total number of stored files is ~330 millions (x3 copies ~= 1 billion
objects)

We customized mogile cluster next way:
- 8 federated db boxes with backup mysql replication
- 2 trackers per db group (16 total)
- 23 storage nodes, each has 8 250-320G drives (one for each group)

Our performance is 
- 50 writes per second per whole cluster (tested up to 500 wps) 
- 120+ reads per second per storage node (~ 3000 reads per second on a
whole
cluster)
- 80% full device recovery (re-replication) usually takes 30 hours with
no affect to production performance.

We developed custom web gui tool and init.d scripts to manage multiple
(8) instances of storage daemons (mogstored) on a given storage node.
GUI tool also comes handy with graphical display of usage pattern and
device/storage node availability. 

In addition, we designed custom nagios checks that trigger notifications
about tracker/storage node availability along with single drive (device)
failure.

I hope you find this helpful.

Oleg


-----Original Message-----
From: mogilefs-bounces at lists.danga.com
[mailto:mogilefs-bounces at lists.danga.com] On Behalf Of Justin Huff
Sent: Thursday, December 13, 2007 9:42 AM
To: Clinton Goudie-Nice
Cc: mogilefs
Subject: Re: Questions about MogileFS in a production environment

I'm not sure we'd count as 'large scale' -- but we use it at Picnik.

We have about 5.7TB, using about 2.7TB.  That's about 6.6million files
with a replcount of 2.

--Justin

Clinton Goudie-Nice wrote:
> Dormando was kind enough to point me at some info in the mailing list
> from a few weeks ago where people talked about the scale of the
MogileFS
> stores they're currently using.
> 
> Could anyone mention some names of some reasonably large scale web
> sites/services which I might recognize that are using MogileFS?
> 
> Also, I've read a bit about trouble serving files greater than 100mb.
Is
> there some limit on file size that I should be aware of?
> 
> Thanks again!
> 
> Clint
> 
> 
> Twelve Horses <http://web.twelvehorses.com/> 	Mobile
> <http://web.twelvehorses.com/info_center/whitepaper/> 	Social
> <http://web.twelvehorses.com/solutions/blogging_for_business/>
Web
> <http://web.twelvehorses.com/solutions/website_design/> 	Email
> <http://web.twelvehorses.com/technology/>
> 
> 
> *Clinton Goudie-Nice*
> Architect / Senior Software Engineer
> cgoudie at twelvehorses.com <mailto:cgoudie at twelvehorses.com>
> Phone: 	+1.801.571.2665 ext 3264
> Mobile: 	+1.801.915.0629
> Fax: 	+1.801.571.2669
> LinkedIn: 	http://www.linkedin.com/in/cgoudie
> 
> 	
> *Twelve Horses*
> 13961 Minuteman Drive
> Suite 125
> Draper, UT 84020
> *www.twelvehorses.com <http://www.twelvehorses.com>*
> 
> 
> 
> 
> 
> On Wed, 2007-12-12 at 16:06 -0700, Clinton Goudie-Nice wrote:
>> Is anyone on the list using mogile in a large scale production
>> environment?
>>
>> I'm considering running MogileFS in a production environment; sort of
>> a giant disk in the sky, for
>> an assortment of Internet apps. It will potentially be serving up
>> terabytes of data, mostly in small image files, probably behind a
>> squid caching proxy.
>>
>> Is there anything people on the list could comment on? How stable is
>> it, how big are your various production stores, etc...
>>
>> Thanks!
>>
>> Clint
>