Memory leak?

[Richard Edward Horner]

Using S3 is much more complicated than writing to a file and there is
much more code involved. I would not be surprised if there were a
memory leak in the S3 code, one of its dependencies or how brackup
interacts with it.

Is this problem a symptom of backing up a bunch of files or a single large file?

Does Amazon have some sort of dev sandbox where you can test this sort
of thing without getting charged for transfer or storage?

Thanks, Rich(ard)

On 8/21/07, robb <robb at canfield.com> wrote:
> I just tried a 1GB file using my SFTP (similar to the file system) with
> block sizes of 64MB and 5MB. In both cases Brackup was stable at between
>   120 MB and 60 MB (approx). The number of files was modest.
>
> Backing up a system with 96,000 files did appear to consume more RAM but
> still only 114 MB with a 5MB block size.
>
> I do not have S3 running yet so I cannot confirm that. But if this only
> happens under S3 then that narrows it down making it easier for Brad
> (the author) or someone else to locate the bug.
>
> David Leadbeater wrote:
> > Hi,
> >
> > I'm getting an out of memory from brackup when backing up some large
> > files to
> > S3. (Well, not *that* large -- the largest file is 250MB).
> >
> > This is under Perl 5.6.2 and 5.8.8 (I tried upgrading Perl just incase)
> > on FreeBSD 5.2. Using Brackup 1.0.5 installed from FreeBSD ports. The
> > box has 1GB RAM.
> >
> > If I watch the brackup process while it's running it does seem to use a
> > lot of memory.. (Around 130MB to 190MB RSS, the VSZ is around 500MB but
> > I guess that's just files being mmap()ed)
> >
> > Eventually it dies with an error like the one below.
> >
> > It seems to carry on where it left off if I start it again so it's
> > fairly easy to work around anyway.
> >
> > Any ideas?
> >
> > Thanks,
> > David
> >
> >   * storing chunk: [xxx(/home)] t=f yyy{off=99614720,len=5242880}
> >   * storing chunk: [xxx(/home)] t=f yyy{off=104857600,len=5242880}
> >   * storing chunk: [xxx(/home)] t=f yyy{off=110100480,len=5242880}
> >   * storing chunk: [xxx(/home)] t=f yyy{off=115343360,len=5242880}
> >   * storing chunk: [xxx(/home)] t=f yyy{off=120586240,len=5242880}
> >   * storing chunk: [xxx(/home)] t=f yyy{off=125829120,len=5242880}
> >   * storing chunk: [xxx(/home)] t=f yyy{off=131072000,len=5242880}
> >   * storing chunk: [xxx(/home)] t=f yyy{off=136314880,len=5242880}
> >   * storing chunk: [xxx(/home)] t=f yyy{off=141557760,len=5242880}
> > Error running backup: Out of memory during "large" request for 8392704
> > bytes, total sbrk() is 529641472 bytes at
> > /usr/local/lib/perl5/site_perl/5.8.8/Brackup/PositionedChunk.pm line 98.
> >
> > Error running backup: Out of memory during "large" request for 4198400
> > bytes, total sbrk() is 529641472 bytes at
> > /usr/local/lib/perl5/site_perl/5.8.8/Brackup/Root.pm line 230, <GEN1>
> > line 1.
> >
> > Use of uninitialized value in numeric eq (==) at
> > /usr/local/lib/perl5/site_perl/5.8.8/Brackup/GPGProcess.pm line 35.
> > Error running backup: size not right at
> > /usr/local/lib/perl5/site_perl/5.8.8/Brackup/GPGProcess.pm line 35.
>
>


-- 
Richard Edward Horner
Engineer / Composer / Electric Guitar Virtuoso
rich at richhorner.com
http://richhorner.com - updated June 28th