conceptual overview?

[Brad Fitzpatrick]

If tracker goes down, no problem... clients will use another.  Then just
restart tracker later.  (or don't)

If storage node goes down, no problem... copies are elsewhere.  If two go
down and your replica count for any file class is only 2 copies, you may
have some files inaccessible until it comes back up.

If a database goes down, your cluster manager should bring another one
up... 5-15 seconds of downtime.  (we plan to add soon slave support, so
you can hang MySQL slaves off the HA master pair, so you at least have
readonly support for the 5-15 seconds while the HA master is switching...)

If all databases goes down, no readonly or rewrite abilities.  (unless the
application has cached the filename->URL mappings externally, as you
should do for hot files, avoiding the tracker altogether.... then you have
read-only access at least for hot files...)

If all databases catch on fire and you have no backups, you're largely
fucked.  You have files all over the network, but you don't know what
their filenames are.  It's akin to recovering an ext2/ext3 filesystem and
having a /lost+found/ directory full of every file in your filesystem.
There's a long-standing request to store filenames beside files on the
filesystem, to make repairs like this possible.  This hasn't been done.

As for configs:

DBs:  reliable storage subsystem.  battery-backed write-back cache
is nice, speeds commits.  big disk cache helps too, and lots of memory on
the machine for InnoDB is nice.  Get two, setup with heartbeat (Linux-HA).

Storage nodes:  Lots of SATA disks, one per mogile device.  3ware
controller or something is good.  Lot of memory helps the buffer cache.
Don't be tricked into seeing "7GB of memory free!" and thinking, "Oh, I
should run more stuff on this box!".  Because that 7GB of memory is being
used for file page cache, which makes Mogile effectively faster.

Trackers:  whatever.  Needs some CPU, but not much.  Run 'em on your first
few storage nodes (2's probably enough, 3 might be overkill....)

Any other questions?

- Brad



On Mon, 15 Jan 2007, Dag Arneson wrote:

> Brad,
>
> Thanks!  That was very helpful.
>
> Lingering questions: what happens when any of these pieces go down?
> Suppose the database server (or the entire database cluster) erupts in
> flames, what happens to the data being stored in mogilefs?  Is there a
> way to recover?  (Of course one would likely have bigger problems if
> that happened, but I'm curious)  What sort of configurations are typical
> for the trackers and storage nodes?  Are they often running on the same
> boxes or what?
>
> Dag
>
> Brad Fitzpatrick wrote:
> > Dag,
> >
> > Check out:
> >
> >    http://mogilefs.schtuff.com/highleveloverview
> >
> > Let me know if I missed anything.  I swear I'd written all that before,
> > but I couldn't find it just now.
> >
> > I'm working on consolidating all the docs into the mogilefsd server code
> > (into POD), and then the POD will autobuild to HTML on the danga.com
> > website (and on search.cpan.org).
> >
> > - Brad
> >
> >
> > On Fri, 12 Jan 2007, Dag Arneson wrote:
> >
> >> Hi,
> >> I'm looking into using mogilefs, and while there seem to be a number of
> >> howtos out there, I'm looking for a document to bootstrap my
> >> understanding of how mogilefs works, bridging the gap between the
> >> high-level overview and low-level setup/API docs.  Something with
> >> diagrams or at least that would make sense to put diagrams into would be
> >> ideal.  Can someone point me in the right direction?
> >>
> >> Thanks!
> >>
> >> --
> >>   Dag Rorek Arneson
> >>   JanRain Inc.
> >>   OpenID Identifiers: rorek.org, dag.myopenid.com
> >>   irc.freenode.net: rorek in #openid
> >>
> >>
> >
> >
>
>