chunks?

Lamont Granquist lamont at scriptkiddie.org
Wed May 30 17:55:19 UTC 2007


Does MogileFS chunk storage like GoogleFS does?  One of the strengths of 
GoogleFS's design is that it writes out storage in 64MB chunks on the 
chunkserver.  If you've got thousands of little 10kB files they'll get 
coalesced into 64MB chunk files.  When a fileserver needs to be replicated 
the chunkfiles can be replicated which means you can stream 64MB at a 
time.  A back of the envelope calculation of the total time to stream 64MB 
is:

4 ms + 64 MB / 30 MBs ~= 2 seconds

versus unchunked seeking every 10kB:

6553 * 4 ms + 64 MB / 30 MBs ~= 26 seconds

So how does MogileFS do this?  If I fill up 2TB with 10kB blobs of data 
under MogileFS how long does it take to get all that data off the disk? 
Does it have to do a seek for every 10kB?


More information about the mogilefs mailing list