mogtool storing large file:

[Greg Connor]

Hello all,

I wrote earlier with questions about mogstored dying, and someone  
suggested that I should replace my old RedHat 8 machines with a newer  
build.

I've rebuilt the cluster (16 nodes) using CentOS5.1 and the results  
were a bit better at first.  But on the first real large-file test, I  
still get problems with some of the chunks not being saved properly,  
and mogtool is not able to reliably store large files for me.



I would love it if someone could tell me if I'm just doing something  
wrong, if I have some sort of hidden problem, or if mogtool is just  
not complete, obsolete or what.  Some of the files I need to store are  
30G (definitely larger than the RAM on the client or storage nodes.

If mogtool is known to have problems, what other options are there for  
dealing with very very large files?  I'd really like mogile for this  
but I'd rather not write my own scheme for handling large files (these  
would be like a 1.6 TB directory as a tar stream, or a number of 30G  
files)... Is mogile just not a good option here? should I start  
evalulating other solutions like hapdoop?


Here's the error generated by mogtool.


WARNING: Unable to save file 'collect-20080513-vol9,2201': Unable to  
create new file at /usr/bin/mogtool line 812, <Sock_minime336:7001>  
line 11907.
MogileFS backend error message: unknown_key unknown_key
System error message: Unable to create new file at /usr/bin/mogtool  
line 812, <Sock_minime336:7001> line 11907.
This was try #9707 and it's been 10578.37 seconds since we first  
tried.  Retrying...

WARNING: Unable to save file 'collect-20080513-vol9,2203': Unable to  
create new file at /usr/bin/mogtool line 812, <Sock_minime336:7001>  
line 11907.
MogileFS backend error message: unknown_key unknown_key
System error message: Unable to create new file at /usr/bin/mogtool  
line 812, <Sock_minime336:7001> line 11907.
This was try #9705 and it's been 10572.94 seconds since we first  
tried.  Retrying..
multi/IO/Handle.pm line 399

MogileFS backend error message: unknown_key unknown_key
System error message: MogileFS::NewHTTPFile: unable to write to any  
allocated storage node at /usr/lib64/perl5/5.8.5/x86_64-linux-thread- 
multi/IO/Handle.pm line 399

This was try #1 and it's been 10.79 seconds since we first tried.   
Retrying...

w