large file investigation..

[Eric Lambrecht]

This is a bit of a silly question, as I'm about to figure out the answer
for myself... but just in case somebody is around with an answer or
hint...

Our mogile setup has difficulties replicating large files between nodes
and storing large files. I'd say the big files we have problems with are
all over 50MB at least (I don't have an actual number yet, but they
range from 5MB to 500MB). The replication process is unable to
completely copy files between nodes on maybe 50% of its tries, and when
we store files things fail maybe 50% of the time.

Luckily, after retrying a couple times, things always ultimately work.

I'd move to the chunking of files, but we need the request range stuff.

So my question is this: does anybody know, fundamentally, why mogile has
such a hard time writing really big files? My first glance of the code
didn't lead me to think there should be any problem for writing... We've
found and successfully tweaked some of the code for reading files that
likes to read the entire contents of files into RAM before returning
them.

I'll be working on this all week. Our code is maybe 3 weeks out of date
with CVS, and I'll fix that first to see if things improve.

and unrelated to this - I wrote a Java client API for mogile. I haven't
had a chance to package it with a sexy build script or documentation
yet, but if you're really hankering for something like this now, email
me and I'll send it to you. Otherwise I'll throw it on the web sometime
this week or next.

Eric...