Offsite Replication

[Eric Lambrecht]

I think you'd actually do better to expose an interface for getting 
notifications of newly stored files and newly deleted files for the 
purpose of archiving, rather than trying to treat some storage node as a 
special 'storeonly' node.

It would then be almost trivial to make a little plugin or worker that 
watches those notifications and copies new files to S3 (or whatever) and 
deletes old ones.

We're trying to duplicate our mogile pool (while it is running) so we 
can split apart two applications that depend on it. With this 
notification, we could make a copy of our live mogile pool, then have a 
worker keep an eye on updates to make sure the copy stays up to date 
until we're ready to split up our application code to use one pool or 
the other.

Eric...



Brad Fitzpatrick wrote:
> You should be able to write a ReplicationPolicy subclass for it.
> 
> The perhaps non-existent part is marking a device as "alive to get new
> files from the replicator, but not alive enough to serve traffic".  We
> have the opposite of that recently (drain), which serves traffic, but
> doesn't get new files, and adding drain supported made us abstract out all
> the state stuff, so adding new states is easy.  Wonder what we'd call that
> state... so far we have alive, down, dead, readonly, drain ...
> 
> The other option is doing backups outside of MogileFS, just copying FID #1
> offsite, then #2, then #3... since it's all sequential, your backup
> software can resume where it left off quite easily.
> 
> I'd love to get the "get files but don't serve from them" support into the
> core, though.  Seems useful.
> 
> 
> On Fri, 22 Jun 2007, Erik Osterman wrote:
> 
>> We would like to have files replicated automatically to offsite devices.
>> These offsite devices would not and should not be used by MogileFS
>> clients;  they should only be used as a remote backup. Has it been
>> suggested or is there otherwise a way to accomplish this with the
>> current (svn) version of MogileFS?
>>
>> Best,
>>
>> Erik Osterman
>>
>>
>>
>>