code restructure; svn a little unstable

[Brad Fitzpatrick]

On Wed, 7 Jun 2006, Andy McFarland wrote:

> On Wed, 2006-06-07 at 00:12 +0000, Brad Fitzpatrick wrote:
> > Mogilers,
>
> Mogilerator!
>
> > Notice that last file?  Yes, replication policy is now abstracted out, so
> > people can do bizarre policies that fit their site-local biz rules/etc,
> > without dirtying the core.
>
> Excellent stuff, looking interesting.
>
> When I added our filthy policy, I went for "soft" rules - if you can't
> find an ideal device for replication, complain and replicate anyway (but
> always honour the deep-down hard rules about never replicating a file to
> the same host). I figured replication outside the policy was better than
> no replication. I suppose it's up to a policy if it makes "hard" or
> "soft" rules.
>
> ...but the problem with soft rules is you end up wanting to shuffle
> things around after your resources have changed. Or you could imagine a
> policy "distribute this class to as many hosts as possible", where the
> ideal changes the more hosts you bring up. A similar thing happens at
> the moment if you let your mogile hosts fill before plugging a new one
> in. All the writes go to the new host.
>
> So I'm wondering if there's a place for a "shuffle" task like a
> replicate, even if the mindev count has been met?

I'm liking it.  I've wanted the same.

So the replicator policy can return "replicate to devid 6, but I'm not
happy about it".  Mogile will do it, not the unhappiness, and return
later in the shuffle task, asking the replicator how it actually wants
things laid out.

> Also - with bunches of mogile hosts on different networks, we've ended
> up with "put" and "get" policies as well as "replicate". I've got
> certain lightweight files I generate from other files, I only ever want
> these put to a certain network (I never replicate them). The get policy
> might be more universal however - "prefer hosts on my local network".

I'll work on making that a policy too.

I also have to pass more things to the replicator policy class:  like the
file name... maybe your policy involves parsing the format of that and
having file-specific rules.  Although then you should probably use a
different file class.

Anyway, loving the suggestions.... keep 'em coming.

- Brad