Looking for general scalability insights

[Nathan Schmidt]

To echo Dormando's point, make sure you use memcache for your
path-to-fid mapping, either at the tracker level or in your
application. We do the latter, since we were writing our data layer
from scratch it was simple to plug that part in.

In our experience, the size of the files is less important than the
number of files you're dealing with. We have 35 million files tracked
on a single mysql master-master pair for the database, plus 2 tracker
instances. That's on commodity 2Ghz C2D 1/2 U machines with 6Gb ram.
The sysload of those database boxes never goes above 0.2, and disk
utilization right now during our peak serving and editing hours is 4%.

Pain points come in when you lose or remove a mogstored instance and
the system needs to rebalance. We've seen up to 10 hours of thrashing
when drives go bad, which leads to mildly degraded performance,
particularly for writes. NetApp probably handles a little better but
you can't beat the price and openness of MogileFS.

Have fun,
-Nathan


On Nov 8, 2007 10:15 AM, dormando <dormando at rydia.net> wrote:
>
> Yoav Steinberg wrote:
> > Hello,
> > I'm looking for some information (or personal experience stories) about
> > how scalable mogile is. Is it realistic to handle lets say 100 storage
> > nodes with 20tera storage connected to each and ten's of millions of files?
>
> Cache your paths, and it'll work fine ;) I can't post numbers from what
> I've ran, but they've been bigger than what you describe, with a single
> database (pair) as a backend.
>
> I personally believe it does better with devices being more spread out,
> but that might just be me. I know other people have even larger instances.
>
> The idea is that you can add storage nodes, frontend perlbals, and
> trackers all willy-nilly and it won't slow down. There are a couple pain
> points on the tracker and the database isn't scalable past one yet (but
> might support read-only slaves fine). If you hit those pain points,
> maybe you can contract one of us to fix it for ya real quick ;)
>
> -Dormando
>