Dropping mode optimization from Metafile?

[Richard Edward Horner]

I'd say bring this up again when Brad gets back from camping or
whatever because he probably has reasons for this architecture but I
do agree that the issues you point out are unfortunate.

Thanks, Rich(ard)

On 8/28/07, robb <robb at canfield.com> wrote:
> During a test of a medium scale backup (simulating RAM and directory
> overhead for 77,000 files averaging 1 MB) I am finding some higher than
> desired RAM usage.
>
> Brackup buffers all of the file stats in RAM (something that is a bit
> tricky to change). This is a constant for the run, but Brackup also
> buffers all chunk data until the backup is done and the metafile
> written. This chunk data can add up to a considerable amount of usage.
> The main reason for buffering chunk data seems to be to optimize the
> size of the file by remembering the most common file and directory
> modes. If the file/dir is the default mode then the mode line need not
> be added.
>
> While optimizing the size of a file is always nice, the downsides for
> large backups may overshadow this:
>
> - A failure during backup leaves then brackup metafile out of date (or
> non-existent). Even if MANY files are properly saved,, the index to them
> is not available.
>
> - RAM is consumed and CPU resources used for saving this data in RAM.
> This becomes an issue on virtualized systems with low RAM thresholds OR
> for any system with LARGE data sets. Brackup currently stores the entire
> file list in RAM, but during backup the chunk references are added to
> this and it is those chunk references that start to add up for large
> files or large backup sets and small a small chunk size makes it even worse.
>
> - The optimization is relatively small (actually very small) size wise
> given the other data in the file.
>
> I vote to drop the optimization file size and favor RAM (and execution
> speed) by streaming the metafile as each file is finished. Any thoughts?
>
>


-- 
Richard Edward Horner
Engineer / Composer / Electric Guitar Virtuoso
rich at richhorner.com
http://richhorner.com - updated June 28th