large file investigation..

[Brad Fitzpatrick]

Eric,

Sorry I can't be more help at the moment, but let me know what you find
and I'll be more than happy to merge your patches.

If I get some time this week, I can look into it, but I imagine you'll
beat me to it.

I never designed it for large files, so it's quite likely I do something
dumb somewhere.  My plan for large files was always to build smarts into
the client, where a large file is really just dozens/hundreds of
medium-sized mogile fid.  Never totally got around to it, but you can see
how mogtool imports huge files like that, so we started.

- Brad


On Sat, 1 Oct 2005, Eric Lambrecht wrote:

> This is a bit of a silly question, as I'm about to figure out the answer
> for myself... but just in case somebody is around with an answer or
> hint...
>
> Our mogile setup has difficulties replicating large files between nodes
> and storing large files. I'd say the big files we have problems with are
> all over 50MB at least (I don't have an actual number yet, but they
> range from 5MB to 500MB). The replication process is unable to
> completely copy files between nodes on maybe 50% of its tries, and when
> we store files things fail maybe 50% of the time.
>
> Luckily, after retrying a couple times, things always ultimately work.
>
> I'd move to the chunking of files, but we need the request range stuff.
>
> So my question is this: does anybody know, fundamentally, why mogile has
> such a hard time writing really big files? My first glance of the code
> didn't lead me to think there should be any problem for writing... We've
> found and successfully tweaked some of the code for reading files that
> likes to read the entire contents of files into RAM before returning
> them.
>
> I'll be working on this all week. Our code is maybe 3 weeks out of date
> with CVS, and I'll fix that first to see if things improve.
>
> and unrelated to this - I wrote a Java client API for mogile. I haven't
> had a chance to package it with a sexy build script or documentation
> yet, but if you're really hankering for something like this now, email
> me and I'll send it to you. Otherwise I'll throw it on the web sometime
> this week or next.
>
> Eric...
>
>
>