Memory leak?

Richard Edward Horner rich at richhorner.com
Tue Aug 21 21:34:48 UTC 2007


Ohh, the other thing I meant to ask David is if he was encrypting his
data with GPG or not.

Thanks, Rich(ard)

On 8/21/07, robb <robb at canfield.com> wrote:
> Yea, I just looked at the S3 code. Much more complex. I am activating my
> S3 account today. I will do some testing. My company is donating the
> resources. At 0.46/GB round trip (1 gb saved, uploaded, downloaded) the
> cost is very minimal.
>
> S3 is critical to one of the jobs destined for Brackup so might as well
> get to it.
>
> Robb
>
> Richard Edward Horner wrote:
> > Using S3 is much more complicated than writing to a file and there is
> > much more code involved. I would not be surprised if there were a
> > memory leak in the S3 code, one of its dependencies or how brackup
> > interacts with it.
> >
> > Is this problem a symptom of backing up a bunch of files or a single large file?
> >
> > Does Amazon have some sort of dev sandbox where you can test this sort
> > of thing without getting charged for transfer or storage?
> >
> > Thanks, Rich(ard)
> >
> > On 8/21/07, robb <robb at canfield.com> wrote:
> >> I just tried a 1GB file using my SFTP (similar to the file system) with
> >> block sizes of 64MB and 5MB. In both cases Brackup was stable at between
> >>   120 MB and 60 MB (approx). The number of files was modest.
> >>
> >> Backing up a system with 96,000 files did appear to consume more RAM but
> >> still only 114 MB with a 5MB block size.
> >>
> >> I do not have S3 running yet so I cannot confirm that. But if this only
> >> happens under S3 then that narrows it down making it easier for Brad
> >> (the author) or someone else to locate the bug.
> >>
> >> David Leadbeater wrote:
> >>> Hi,
> >>>
> >>> I'm getting an out of memory from brackup when backing up some large
> >>> files to
> >>> S3. (Well, not *that* large -- the largest file is 250MB).
> >>>
> >>> This is under Perl 5.6.2 and 5.8.8 (I tried upgrading Perl just incase)
> >>> on FreeBSD 5.2. Using Brackup 1.0.5 installed from FreeBSD ports. The
> >>> box has 1GB RAM.
> >>>
> >>> If I watch the brackup process while it's running it does seem to use a
> >>> lot of memory.. (Around 130MB to 190MB RSS, the VSZ is around 500MB but
> >>> I guess that's just files being mmap()ed)
> >>>
> >>> Eventually it dies with an error like the one below.
> >>>
> >>> It seems to carry on where it left off if I start it again so it's
> >>> fairly easy to work around anyway.
> >>>
> >>> Any ideas?
> >>>
> >>> Thanks,
> >>> David
> >>>
> >>>   * storing chunk: [xxx(/home)] t=f yyy{off=99614720,len=5242880}
> >>>   * storing chunk: [xxx(/home)] t=f yyy{off=104857600,len=5242880}
> >>>   * storing chunk: [xxx(/home)] t=f yyy{off=110100480,len=5242880}
> >>>   * storing chunk: [xxx(/home)] t=f yyy{off=115343360,len=5242880}
> >>>   * storing chunk: [xxx(/home)] t=f yyy{off=120586240,len=5242880}
> >>>   * storing chunk: [xxx(/home)] t=f yyy{off=125829120,len=5242880}
> >>>   * storing chunk: [xxx(/home)] t=f yyy{off=131072000,len=5242880}
> >>>   * storing chunk: [xxx(/home)] t=f yyy{off=136314880,len=5242880}
> >>>   * storing chunk: [xxx(/home)] t=f yyy{off=141557760,len=5242880}
> >>> Error running backup: Out of memory during "large" request for 8392704
> >>> bytes, total sbrk() is 529641472 bytes at
> >>> /usr/local/lib/perl5/site_perl/5.8.8/Brackup/PositionedChunk.pm line 98.
> >>>
> >>> Error running backup: Out of memory during "large" request for 4198400
> >>> bytes, total sbrk() is 529641472 bytes at
> >>> /usr/local/lib/perl5/site_perl/5.8.8/Brackup/Root.pm line 230, <GEN1>
> >>> line 1.
> >>>
> >>> Use of uninitialized value in numeric eq (==) at
> >>> /usr/local/lib/perl5/site_perl/5.8.8/Brackup/GPGProcess.pm line 35.
> >>> Error running backup: size not right at
> >>> /usr/local/lib/perl5/site_perl/5.8.8/Brackup/GPGProcess.pm line 35.
> >>
> >
> >
>
>


-- 
Richard Edward Horner
Engineer / Composer / Electric Guitar Virtuoso
206.856.2808 - rich at richhorner.com
http://richhorner.com - updated June 28th


More information about the brackup mailing list