Using large files with MogileFS

[Benjamin James Meynell]

I am experiencing the same problem. The only difference for me is that mogstored crashes for files ~180MB and above, but I get the same error(s) so I'm assuming the crash threshold is due to memory differences (the machine I'm testing on has ~650MB of free RAM). When running mogstored in the foreground it quites with an "Out of Memory!" error. I've also noticed that upon a fresh restart of mogstored it is more forgiving and can accept files it previously ran out of memory on. Was a solution ever found? 

>   crash log: Negative length at
>   /usr/share/perl5/Danga/Socket.pm line 1133.
>
>   Socket.pm states (line 1127-1142):
>       # if this is too high, perl quits(!!).  reports
>   on mailing lists
>       # don't seem to point to a universal answer. 
>   5MB worked for some,
>       # crashed for others.  1MB works for more
>   people.  let's go with 1MB
>       # for now.  :/
>       my $req_bytes = $bytes > 1048576 ? 1048576 :
>   $bytes;
>
>       my $res = sysread($sock, $buf, $req_bytes, 0);
>       DebugLevel >= 2 && $self->debugmsg("sysread =
>   %d; \$! = %d", $res, $!);
>
>       if (! $res && $! != EWOULDBLOCK) {
>           # catches 0=conn closed or undef=error
>           DebugLevel >= 2 && $self->debugmsg("Fd \#%d
>   read hit the end of the road.", $self->{fd});
>           return undef;
>       }
>
>       return \$buf;
>
>   dormando schreef:
>
>     Can you get the crashlog from mogstored?
>
>     Run it in the foreground or watch syslog/etc.
>
>     -Dormando
>
>     Jaco wrote:
>
>       Hi,
>
>       MogileFS is running fine now, but there are
>       still some major problems when I try to upload
>       files bigger than 1 GB. With such large files I
>       get errors like "Send failure: Connection reset
>       by peer" and "select/poll returned error" and
>       the mogstored daemon is suddenly killed. Is this
>       normal?
>
>       If it is normal, is there a way to automatically
>       split large files into smaller chunks during
>       upload using mogilefsd? I know this can be done
>       with mogtool, but I'm looking for a more
>       transparent method within mogilefsd.
>
>       Thanks in advance.