Save large file problem

Yuri Subach subach at whirix.com
Mon Sep 5 02:54:02 PDT 2005


Hello Brad,

As you suggested, I split large files into chunks. MogileFS works really good. 
I'm wondering about optimal size of the chunk. 10MB size works fine, but I'm 
not sure that system will be stable. When I tried 100MB mogstored show this error:

epoll() returned fd 11 w/ state 1 for which we have no mapping.  removing.

Can you tell me how to determine largest possible chunk size? In other words, 
where is the bottle neck of mogstored?

Brad Fitzpatrick wrote:
> It shouldn't be trying to get it all into memory.  I don't recall any part
> of MogileFS in particular that would try to do that.
> 
> But you are pushing what I'd consider reasonable sizes for MogileFS.  Big
> files should be broken up into chunks.  (mogtool can help you out with
> that)  I'd love it if MogileFS did the auto-chunking for you
> transparently, but I haven't gotten around to that (or had a need).
> 
> - Brad
> 
> 
> On Fri, 2 Sep 2005, Yuri Subach wrote:
> 
> 
>>Hi,
>>
>>I tried to put large file (650MB) into MogileFS storage, but mogstored die
>>with "Out of memory!" message. Machine has 256 MB. My questions: is it trying
>>to put whole file in memory before saving to disk?
>>
>>Can someone advise how to use MogileFS with large files?
>>
>>--
>>Yuri Subach  subach at whirix.com
>>Director of Whirix Ltd.  http://www.whirix.com
>>
>>

-- 
Yuri Subach  subach at whirix.com
Director of Whirix Ltd.  http://www.whirix.com


More information about the mogilefs mailing list