mogilefs and geographic redundancy

[Brad Fitzpatrick]

My problem with this patch is that it continues to do something which I
did wrong in the first place:

Put policy into the core.

Everybody I talk to has a different site-local policy on how they
need/want/think they need/want things replicated.  None of them are
subsets of some master superset that we're going to find.

What we need to do is make a nice plugin setup so you can have per-class
policies.

So you can say, "For the App1/thumbnails class, use
MogileFS::Policy::Geographic".

Then the MogileFS::Policy::Geographic object is instantiated from the
domain/class, and whenever the MogileFS core needs to make a decision:

   -- where/if to replicate after all new files or disk failures.

   -- give a client and its request, and all devices a file is on,
      how to prioritize their order returned.

Then we write a shitload of policies and document then and people can
choose what works for them.

Things like "min_replica_count", which is all we have now per-class, is
just a parameter to any policy that's instantiated.  Or the policy can
ignore it.

I also then want to allow any metadata on devices/hosts, so the policy
plugins have access to that.  So you can mark which city/country your
hosts are in, what speed your devices are, etc.

Does this sound sane?



On Wed, 26 Apr 2006, Andy McFarland wrote:

> On Mon, 2006-04-24 at 11:33 -0600, Jeff Lewis wrote:
> > We basically have 2 datacenters that are in different locations.  Each
> > is supposed to be an exact copy of the other at any given time - within
> > reason.  I can see running MogileFS in each of the 2 locations, but how
> > then, do I keep them in sync?
>
> We also run MogileFS on more than one network, attached is a very simple
> patch that helps us.
>
> We wanted to make mogile smarter about three things when working with
> multiple networks:-
>
>  * Clients getting a file should get a path on their local network if
> possible
>  * Clients putting a file should be able to request which network it
> ends up on
>  * The replicator should put the destination copy on a different network
> to the source copy
>
> The first is easy. I modified our client so after it had called
> mogile_paths it filtered them looking for ones on the local network.
>
> To make the second and third happen I noticed that both
> QueryWorker::cmd_create_open and Mgd::replicate both call
> Mgd::find_deviceid to work out which device to use as a destination.
>
> I tweaked find_deviceid to accept parameters for "try and use devices on
> these networks as a destination" and "try and avoid these networks as
> destinations". I changed replicate to work out which networks to avoid,
> then I changed create_open and the perl api to let clients request which
> networks content ends up on after a MogileFS::new_file.
>
> mogilefsd gets a new command line parameter "--network_replicate" to
> turn on the new replicate behaviour.
>
> When find_deviceid can't find an ideal device, it will complain via the
> mogile console and fall back on the old "probably pick the most empty
> device" behaviour. Replicating to the same network is better than not
> replicating at all. cmd_create_open in particular will try and get 3
> devices, even though it typically only uses the first. If you don't have
> good devices on at least 3 hosts on your ideal network, you'll get
> spurious warnings.
>
> It's all very simple and there are some other things to watch out for
> when mogiling across multiple networks, but this mail is nearly longer
> than the patch, so I'll stop now.
>
> Cheers,
> Andy.
>
>