brackup possible memory leak - consumes 1.5GB+ with 64MB chunk size

[William J. Coldwell]

load averages:  2.03,  2.00,  1.71;               up 38+14:12:15        
21:36:49
47 processes: 46 sleeping, 1 on CPU
CPU states:  0.0% user,  3.8% nice,  0.2% system,  0.6% interrupt,  
95.4% idle
Memory: 588M Act, 287M Inact, 6340K Wired, 12M Exec, 259M File, 1688K  
Free
Swap: 1153M Total, 1110M Used, 43M Free
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU  
COMMAND
     0 root     123    0     0K   22M physiod  140:45  0.00%  0.00%  
[system]
   136 root      85    0    42M 4644K select    29:03  0.00%  0.00%  
named
  4818 billc     85    0  8712K    4K select     7:05  0.00%  0.00% sshd
   303 bb        85    0  2840K  480K socket     2:48  0.00%  0.00% bbd
  1062 root      80   10   789M  181M biowait    2:47  0.00%  0.00%  
perl   <---
  7749 root      80   10   643M  424M biowait    2:02  0.00%  0.00%  
perl   <---
   362 root      85    0  5492K 5688K pause      1:38  0.00%  0.00% ntpd
   233 root      85    0  9668K  536K select     1:07  0.00%  0.00%  
httpd
   476 root      85    0  5944K    4K select     0:40  0.00%  0.00%  
<sshd>
   114 root      85    0  2932K  356K kqueue     0:23  0.00%  0.00%  
syslogd
  1357 root      85    0  4808K  480K kqueue     0:16  0.00%  0.00%  
master
  1278 root      85    0  2900K  356K nanoslp    0:07  0.00%  0.00% cron
   375 postfix   85    0  4808K    4K kqueue     0:07  0.00%  0.00%  
<qmgr>
     1 root      85    0  2932K    4K wait       0:05  0.00%  0.00%  
<init>
  5115 billc     85    0  8712K 1040K select     0:02  0.00%  0.00% sshd
  1507 billc     85    0  2964K 1108K select     0:01  0.00%  0.00%  
screen-4.0.3
   315 bb        85    0  2832K    4K nanoslp    0:01  0.00%  0.00%  
<bbrun>
14993 billc     43    0  2984K 1324K CPU        0:00  0.00%  0.00% top
20831 www       85    0  9668K 2060K semwait    0:00  0.00%  0.00% httpd
19539 www       85    0  9668K 2056K semwait    0:00  0.00%  0.00% httpd
20787 billc     85    0  1972K 1760K wait       0:00  0.00%  0.00% bash


I caught it just right before UVM killed it for eating all of the swap.
I believe it's in biowait because of swapping.  This is with current
svn.  Is there anyone willing to add more verbosity to brackup, while
it is running with regards to memory/cpu usage? (ctrl-t wasn't all that
useful).

--
William J. Coldwell "Cryo" WC25-ARIN
Warped, Inc.    http://warped.com
Sr. Custodial Engineer 20BOVINE29
+33° 58' 16.20", -83° 55' 50.04"


On Jul 24, 2009, at 7:36 PM, William J. Coldwell wrote:

> usemymalloc=n, here
>
>
> --
> William J. Coldwell "Cryo" WC25-ARIN
> Warped, Inc.    http://warped.com
> Sr. Custodial Engineer 20BOVINE29
> +33° 58' 16.20", -83° 55' 50.04"
>
>
> On Jul 24, 2009, at 4:03 PM, Shaun Amott wrote:
>
>> On Tue, Jul 21, 2009 at 09:33:34AM +0100, George Hills wrote:
>>>
>>> I've been looking at a server which backs up about 600GB of data
>>> overnight from a local disk to Amazon using brackup 1.09.
>>>
>>> The backup process has worked faultlessly for several weeks, but
>>> recently begain failing.
>>>
>>> The pattern to the failures is that brackup consumes all the  
>>> available
>>> RAM and then gets killed. The available RAM on the box is upwards  
>>> of 2GB.
>>>
>>> The data on the local disk changes quite a lot. brackup does not  
>>> always
>>> run out of memory at the same time. Sometimes it seems to run for  
>>> a long
>>> period and complete a substantial amount of the upload, while RAM  
>>> use
>>> remains stable around 400MB, before blowing up.
>>>
>>
>> Is your Perl binary compiled with usemymalloc=y?
>>
>> Shaun
>>
>> -- 
>> Shaun Amott // PGP: 0x6B387A9A
>> "A foolish consistency is the hobgoblin
>> of little minds." - Ralph Waldo Emerson