Offsite Replication

Nathan Schmidt nathan at
Mon Jun 25 21:02:18 UTC 2007

Being impatient, we're about to deploy an internal tool to do Mogile- 
to-S3 following. Right now we're pulling fids from a db slaved off of  
the primary stores, and pushing files to s3 for archive/disaster  
recovery. As long as you're ok with not keeping track of deletes it's  
pretty simple. It's not going to end up using the trackers or plugin  
api for the time being (particularly since we're a little behind SVN  
head) but that may make more sense for us in the future. We'll figure  
out a way to share this with others once we've tested a bit more.


On Jun 25, 2007, at 1:22 PM, Jared Klett wrote:

> It's really cool to see this subject resurrected - I believe it was
> first brought up a while ago with a discussion of  support for  
> Amazon S3
> as a "store but don't serve" device.
> I would call such a device state "storeonly" or "writeonly" or some
> non-compound word. :)
> -----Original Message-----
> From: mogilefs-bounces at
> [mailto:mogilefs-bounces at] On Behalf Of Brad  
> Fitzpatrick
> Sent: Monday, June 25, 2007 4:19 PM
> To: Erik Osterman
> Cc: mogilefs at
> Subject: Re: Offsite Replication
> You should be able to write a ReplicationPolicy subclass for it.
> The perhaps non-existent part is marking a device as "alive to get new
> files from the replicator, but not alive enough to serve traffic".  We
> have the opposite of that recently (drain), which serves traffic, but
> doesn't get new files, and adding drain supported made us abstract out
> all the state stuff, so adding new states is easy.  Wonder what we'd
> call that state... so far we have alive, down, dead, readonly,  
> drain ...
> The other option is doing backups outside of MogileFS, just copying  
> #1 offsite, then #2, then #3... since it's all sequential, your backup
> software can resume where it left off quite easily.
> I'd love to get the "get files but don't serve from them" support into
> the core, though.  Seems useful.
> On Fri, 22 Jun 2007, Erik Osterman wrote:
>> We would like to have files replicated automatically to offsite
> devices.
>> These offsite devices would not and should not be used by MogileFS
>> clients;  they should only be used as a remote backup. Has it been
>> suggested or is there otherwise a way to accomplish this with the
>> current (svn) version of MogileFS?
>> Best,
>> Erik Osterman

More information about the mogilefs mailing list