Using large files with MogileFS
Andy Lo A Foe
andy.loafoe at gmail.com
Thu Apr 24 06:32:26 UTC 2008
Hi,
In my experience if you want to store large files in MogileFS (>
100MB) you should definitely be using WebDAV on the storage nodes
(apache, lightttpd or nignix).
Gr,
Andy
On Wed, Apr 23, 2008 at 11:32 PM, Benjamin James Meynell
<bmeynell at colorado.edu> wrote:
> I am experiencing the same problem. The only difference for me is that mogstored crashes for files ~180MB and above, but I get the same error(s) so I'm assuming the crash threshold is due to memory differences (the machine I'm testing on has ~650MB of free RAM). When running mogstored in the foreground it quites with an "Out of Memory!" error. I've also noticed that upon a fresh restart of mogstored it is more forgiving and can accept files it previously ran out of memory on. Was a solution ever found?
>
> > crash log: Negative length at
> > /usr/share/perl5/Danga/Socket.pm line 1133.
> >
> > Socket.pm states (line 1127-1142):
> > # if this is too high, perl quits(!!). reports
> > on mailing lists
> > # don't seem to point to a universal answer.
> > 5MB worked for some,
> > # crashed for others. 1MB works for more
> > people. let's go with 1MB
> > # for now. :/
> > my $req_bytes = $bytes > 1048576 ? 1048576 :
> > $bytes;
> >
> > my $res = sysread($sock, $buf, $req_bytes, 0);
> > DebugLevel >= 2 && $self->debugmsg("sysread =
> > %d; \$! = %d", $res, $!);
> >
> > if (! $res && $! != EWOULDBLOCK) {
> > # catches 0=conn closed or undef=error
> > DebugLevel >= 2 && $self->debugmsg("Fd \#%d
> > read hit the end of the road.", $self->{fd});
> > return undef;
> > }
> >
> > return \$buf;
> >
> > dormando schreef:
> >
> > Can you get the crashlog from mogstored?
> >
> > Run it in the foreground or watch syslog/etc.
> >
> > -Dormando
> >
> > Jaco wrote:
> >
> > Hi,
> >
> > MogileFS is running fine now, but there are
> > still some major problems when I try to upload
> > files bigger than 1 GB. With such large files I
> > get errors like "Send failure: Connection reset
> > by peer" and "select/poll returned error" and
> > the mogstored daemon is suddenly killed. Is this
> > normal?
> >
> > If it is normal, is there a way to automatically
> > split large files into smaller chunks during
> > upload using mogilefsd? I know this can be done
> > with mogtool, but I'm looking for a more
> > transparent method within mogilefsd.
> >
> > Thanks in advance.
>
More information about the mogilefs
mailing list