mogilefs and geographic redundancy

Fred Moyer fred at redhotpenguin.com
Sat Apr 29 03:42:29 UTC 2006


Brad Fitzpatrick wrote:
> My problem with this patch is that it continues to do something which I
> did wrong in the first place:
>
> Put policy into the core.
> 
> Everybody I talk to has a different site-local policy on how they
> need/want/think they need/want things replicated.  None of them are
> subsets of some master superset that we're going to find.
> 
> What we need to do is make a nice plugin setup so you can have per-class
> policies.
> 
> So you can say, "For the App1/thumbnails class, use
> MogileFS::Policy::Geographic".

# approximation based on my Apache config knowledge
<FileType *.thumbnail>
   PolicyHandler Geographic
</FileType>

> Then the MogileFS::Policy::Geographic object is instantiated from the
> domain/class, and whenever the MogileFS core needs to make a decision:
>    -- where/if to replicate after all new files or disk failures.
>    -- give a client and its request, and all devices a file is on,
>       how to prioritize their order returned.

The core should ask the MofileFS::Policy layer for the decision right? 
(assuming that you mean that with the Geographic subclass)

Wrt priorizing the order, here's one to throw away, for what it's (not) 
worth.  Note that this is entirely off the top of my head, and may not 
correspond to the existing MogileFS API (mogilefs has been stable and I 
haven't visited it in a while).  Also note I'm still grokking the 
problemspace.  So this may fail - miserably.  But it's Friday, so it's 
time for faulty thought experiments :)

my $server = MogileFS::Server->new;
$server = handle_request($request);

sub handle_request {

     my $client = $request->client;
     my @policies = $client->server->policies;
	# this server replicates locally, etc
     foreach my $policy (@policies) {
         foreach my $nearest_device
             (sort { $policy->device->nearest_device... } 	
		@{$policy->devices}) {
             if ($nearest_device->has_file($request->file) {
                  $nearest_device->serve_file($request->file);
             }
         }
      }
}

> Then we write a shitload of policies and document then and people can
> choose what works for them.

++

> 
> Things like "min_replica_count", which is all we have now per-class, is
> just a parameter to any policy that's instantiated.  Or the policy can
> ignore it.
> 
> I also then want to allow any metadata on devices/hosts, so the policy
> plugins have access to that.  So you can mark which city/country your
> hosts are in, what speed your devices are, etc.
> 
> Does this sound sane?

Yes it does.


More information about the mogilefs mailing list