brackup possible memory leak - consumes 1.5GB+ with 64MB chunk size

[Gavin Carr]

On Sat, Aug 01, 2009 at 10:15:39PM -0400, William J. Coldwell wrote:
> I wonder if this is during building of the list of files.  Since I'm
> backing up 256k+ files, if this isn't stored in some sort of disk
> backed array, it could suck all of the memory allowed.

Here are some memory stats from a recent run I did, backing up 840k 
files, 2.1TB in size:

  BACKUPS STATS:

  Start Time:                           Wed Jul 29 00:15:18 2009
  End Time:                             Wed Jul 29 15:20:44 2009
  File Discovery Time:                  803s
  Chunk Iterator Time:                  1s
  Chunk Storage Time:                   53504s
  Metafile Storage Time:                18s
  Total Run Time:                       54326s

  Post File Discovery Memory Usage:     895.7 MB
  Post Chunk Iterator Memory Usage:     927.7 MB
  Post Chunk Storage Memory Usage:      3725.6 MB
  Post Metafile Storage Memory Usage:   4266.8 MB
  Peak Memory Usage:                    4266.8 MB

  Number of Files                       840657
  Total File Size                       2135818.7 MB
  Number of Files Uploaded:             2066
  Total File Size Uploaded:             10369.7 MB


So memory usage peaked at 4.3GB in this case, with most of that used
during the chunk storage phase. So looks to me as if that's the place
we should be investigating/optimising.

Cheers,
Gavin