mogstored dying: redux

[dormando]

I'm a little amazed you've been having so many problems :( Big files or 
not I've never really seen this happen. This doesn't happen rarely to 
me, big files or not. Doesn't happen at all.

Is your system/OS still weird? Are you missing huge OS errors with write 
abortions/etc?

Can you run the mogilefsd tracker with the debug level raised, in 
screen, in the foreground, with output redirected to a file? Then 
mail/uploaded it somewhere so we can see?

I'm somewhat highly doubting it's a code problem. The errors really do 
suck a bit, but I can't imagine why you'd be getting that without having 
hardware trouble or really gummed up OS/perl libraries.

-Dormando

> 
> OK that definitely helps.  Lighttpd is back on, and it doesn't look
> like mogstored/lighttpd is dying.  Actually it looks like something
> wrong with the trackers now.
> 
> The various system errors are similar to the ones I've seen before.
> mogtool transfers about 70 chunks successfully and then starts giving
> this error (over and over).
> 
>> MogileFS backend error message: unknown_key unknown_key
>> System error message: Close failed at /usr/bin/mogtool line 816,  
> <Sock_minime336:7001> line 78.
>> This was try #1 and it's been 1.06 seconds since we first tried.   
> Retrying...
> 
> The failed chunks are tried over and over until I kill the job.  Even 
> after killing the mogtool job, I couldn't push any file, with mogtool or 
> with a simple script similar to Mark's test.
> 
> At this point I don't know if mogtool is at fault, but I don't
> actually have any other way of getting large files into the system
> other than rolling my own mogtool, which is likely to be bad for all
> its own reasons and confuse matters further.
> 
> I am also seeing a large number of these errors:
> 
>> System error message: MogileFS::Backend: tracker socket never became 
>> readable (minime336:7001) when sending command: [create_open 
>> domain=dbbackups&fid=0&class=dbbackups-recent&multi_dest=1&key=dwh-20080519-vol9,99 
>> ] at /usr/lib/perl5/site_perl/5.8.5/MogileFS/Client.pm line 268
> 
> Thankfully these seem to be recoverable by mogtool retrying, but they 
> seem to be disturbingly frequent.  I will need to understand what 
> underlying condition is causing this error... I know it's entirely 
> possible that it's a network/machine/storage/whatever problem but I'm 
> not sure where to look for more information or how to start 
> troubleshooting.  I guess this concern applies to all the errors I've 
> seen including:
>  > Close failed at /usr/bin/mogtool line 816
>  > unable to write to any allocated storage node at 
> /usr/lib64/perl5/5.8.5/x86_64-linux-thread-multi/IO/Handle.pm line 399
>  > Connection reset by peer
>  > tracker socket never became readable
>  > socket closed on read at 
> /usr/lib/perl5/site_perl/5.8.5/MogileFS/NewHTTPFile.pm line 335
>  > couldn't connect to mogilefsd backend at 
> /usr/lib/perl5/site_perl/5.8.5/MogileFS/Client.pm line 268
> 
> If they happen rarely, and the tool I'm using can recover, it's not a 
> huge concern, but if they happen frequently enough to raise eyebrows 
> (like 1 transaction in 10) or if they cause an endless loop where we 
> can't recover, then it's a showstopper.
> 
> The general trend here seems to be that errors happen, and MogileFS just 
> plain aborts the current transaction with an unhelpful "die" type of 
> message.  I am reasonably good with perl but I'm not familiar with this 
> code base enough to dive into all the errors listed above and get down 
> to the root cause.  There could be any number of underlying causes, but 
> because error handling is not great, I'm being forced to look at Perl 
> code and guess at root cause instead of going right to the underlying 
> cause.
> 
> Is there some log I should be looking at for more info, or some 
> debugging flag I need to turn on?
> 
> Does anyone else have a winning strategy for dealing with very-large 
> files other than mogtool --bigfile?
> 
> Thanks again for the help and patience.