Writing backup metafile incrementally

[Brad Fitzpatrick]

Good idea.  The patch looks sane just looking at the diff, but I'd like to
look at it in context more first.

Would you mind uploading this to codereview.appspot.com?  They have a
command-line tool you can run from your svn repo that does the upload.

On Tue, Jul 29, 2008 at 11:54 AM, Gavin Carr <gavin at openfusion.com.au>wrote:

> I'm messing around with some pretty big backups at the moment (400-800GB
> trees), and I'm finding that brackup is sucking up huge gobs of RAM while
> doing this (like 8GB).
>
> One of the culprits seems to be the metafile data, which we accumulate
> over the course of the backup and then write out at the end. My metafiles
> are coming in at around 600MB on these backups, and the in-core footprint
> seems about 1GB, so that's a big chunk of ram we're holding for no very
> good reason.
>
> The attached patch writes the metafile incrementally instead, writing to
> a tempfile, and then renaming at the end (so failures don't leave partial
> metafiles lying around). The only wrinkle is that we still have to spool
> entries if we have a CompositeChunk open, because we can't record the
> metafile entry without the chunk checksum.
>
> Does this look sane? Please comment/test.
>
> Cheers,
> Gavin
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.danga.com/pipermail/brackup/attachments/20080801/301ac1c4/attachment.html