Java client [file chunking]

dormando dormando at rydia.net
Tue Oct 30 01:18:04 UTC 2007


I'd suggest testing it.

I've been manually disabling the chunked support, given every use case I 
have requires the files to be streamed from one place to the other. If 
you have a large, contiguous file, you need to re-assemble it in order 
at some point anyway. If you're spoonfeeding clients, you want to start 
and go without having to re-stablish an HTTP session in the middle.

So, if you want to pull data in chunks and can process it in parallel, 
or you want to very evenly fill every storage device, fine... but I 
don't see that happening in any useful way.

It's likely my own lack of imagination here. Someone please prove me 
wrong :)

It feels like a bad default at any rate, since you can't serve the large 
files back like that presently.

-Dormando

Lance Reed wrote:
> I have yet to deploy a large scale setup of mogilefs but have a small 
> test env setup.
> 
> The Java client would fit nicely into our application use.
> Most of my experience so far has been with using mogtool, so I assumed 
> that it was best to chunk files in to 64 MB.
> But I take it from this thread that this is not the case.  Do people 
> usually just put and take files as in without chunking then?
> This would save time on the initial write I assume.  But do little fro 
> the read back.  I have not had a chance to test, but do people see a 
> performance increase when reading back large files that have been 
> chunked due to parallel reads?
> Or is the overhead of re-construction too much to really see a benefit?
> 
> I'd love to hear what folks have been seeing in the real world.
> I am planning to use mogilefs with files that range from 30 KB to 20 GB, 
> and am trying to figure out if I really need to put chunking code in.
> 
> But I could see wanting a patch for the java client to do chunking.
> 
> Thanks for any thoughts on this.
> 
> Lance
> 
> 
> 
> dormando wrote:
>> Curious now...
>>
>> Does anyone use chunked files for anything?
>>
>> I can't think of any reason why you'd get more performance out of it, 
>> and the only benefit being able to stuff files larger than the 
>> individual storage nodes into mogile.
>>
>> Believe it's still the default for mogtool with files > 64M, which 
>> must be confusing for folks?
>>
>> -Dormando
>>
>> Jared Klett wrote:
>>>     We ended up not using the chunked file support for performance
>>> reasons - we just store the whole file and serve it straight off disk.
>>>
>>>     Regardless, I'd be happy to release a patch if there's demand.
>>>
>>> cheers,
>>>
>>> - Jared
>>>
>>
>>
> 



More information about the mogilefs mailing list