[PATCH 3/3] Open files only once

[Gavin Carr]

On Thu, Aug 14, 2008 at 11:12:44AM +0200, Michael Hanselmann wrote:
> On Wed, Aug 13, 2008 at 11:26:14PM +0100, Ed W wrote:
> > I like your direction, but just as interested to hear a little more about 
> > how you are using rsync in your backup strategy?  Is it rsync 3 with the 
> > new incremental mode?
> 
> It depends on the machine. Some have rsync 2, others have rsync 3. rsync
> doesn't consume as much memory because it's written in C (read as: no
> additional language memory overhead). I'm configuring rsnapshot[1] using
> slack[2] to do the work for me.

Just as another datapoint, I've gone from rsnapshot to brackup, rather
the other way around.

I've got to do some reasonable size daily backups, one of a tree with 2.5m
files in it and about 250GB, and one with about 500k files and about 750GB.
We also require both local backups, for quick restores, and offsites, for
disaster recovery.

Our plan was to do rsnapshots overnight to a local backup server, and then
rsync the resultant backup offsite during the day. The rsnapshot leg worked
ok (typically an 8-10 hour backup), but we found that rsync (at least with 
-H to preserve the hard-links, required to keep the storage size manageable)
simply couldn't walk trees of that size fast enough to get the sync done. I
guess if you've got 10 hardlinked copies of a 2.5m file tree, you've got to
check 25m inodes before you can start the sync.

For this particular use case brackup's a much more sane architecture, as it
scales according to the number of unique chunks you've got, rather than the
number of files, and handles incrementals really nicely. Our nightly backups
have dropped from 8-10 hours to 2 hours, and the remote rsync is something 
similar.

So we're now big brackup fans. :-)

As Michael says, though, it is a bit of a memory beast at the moment. I'm
working on trimming that back some as well.

My 2c for the archives.

Cheers,
Gavin