[PATCH 3/3] Open files only once

[Michael Hanselmann]

Hi Ed

On Thu, Aug 14, 2008 at 12:11:26PM +0100, Ed W wrote:
> Did this message reach the mailing list also?  I noticed messages only CC'd 
> to the list don't seem to go there […]

Apparently this mailing list is configured to not send mails again to
people who're already CC'ed.

> rsync3 has an incremental mode which allows it to cut memory usage
> quite considerably.  It relies on a few things coming into alignment
> though so it may not be operating for you anyway? (see the man page).

Yes, I know it has an incremental mode and I've even seen it using it
automatically if you're transferring files between two machines with
rsync 3. However, rsync'ing between version 2 and 3 works fine as well.

> Have you tried snapback2 versus rsnapshot?

I remember seeing its website, but I didn't investigate it further
because it's not included in the two main distributions I'm using these
days, Debian and Ubuntu.

> I like the former because you just setup one cronjob and it rotates
> everything automatically.  No need for multiple manual cronjobs to
> rotate things

rsnapshot also automatically rotates directories, but you need several
cronjobs for different intervals.

> I'm not sure the difference in memory consumption is entirely down to
> language though - I think rsync has simply been written to run
> extremely tightly and a lot of thought given to the datastructures (I
> say this without having looked at the code).

Half a year ago or so I had a look into rsync's source and yes, it's
very optimized. It even keeps common directory prefixes only once for
file paths (at least in some cases).

> I would think it possible for brackup to get within say a factor of 2
> of the consumption of rsync, and perhaps to get close to equalling it
> with some thought to algorithms for large amounts of files? (buffer to
> disk?)

Sure, there are a lot of possibilities for optimizations, but they need
time to implement and the maintainer needs to be convinced. :-)

Regards,
Michael

-- 
http://hansmi.ch/