mogtool storing large file:

[Ask Bjørn Hansen]

On May 14, 2008, at 11:57 PM, Greg Connor wrote:

[Saving large files with mogtool]
> Here's the error generated by mogtool.
>
> WARNING: Unable to save file 'collect-20080513-vol9,2201': Unable to  
> create new file at /usr/bin/mogtool line 812, <Sock_minime336:7001>  
> line 11907.
> MogileFS backend error message: unknown_key unknown_key
> System error message: Unable to create new file at /usr/bin/mogtool  
> line 812, <Sock_minime336:7001> line 11907.
> This was try #9707 and it's been 10578.37 seconds since we first  
> tried.  Retrying...


What does the backend logs say?

There's a bug in the latest released mogtool that makes the automatic  
retrying on failure not work.   Patch below; the change is also in SVN  
(as per r1130).

I forget the details, but I noticed the bug when my backends had some  
sort of configuration problem that made them fail once in a while.

  - ask

--- mogtool	(revision 1129)
+++ mogtool	(revision 1130)
@@ -799,25 +799,26 @@
          my $mogfs = get_mogfs();
          my $dkey = $opts{big} ? "$key,$chunknum" : "$key";

-        # TODO: be resilient to transient errors, retry, etc.
          my $start_time = [ gettimeofday() ];
          my $try = 0;
          while (1) {
              $try++;
-            my $fh = $mogfs->new_file($dkey, $opts{class}, $bufsize);
-            unless (defined $fh) {
-                error("WARNING: Unable to create new file '$dkey'.");
+            eval {
+                my $fh = $mogfs->new_file($dkey, $opts{class},  
$bufsize);
+                unless (defined $fh) {
+                    die "Unable to create new file";
+                }
+                $fh->print($chunkbuf{$cn});
+                unless ($fh->close) {
+                    die "Close failed";
+                }
+            };
+            if (my $err = $@) {
+                error("WARNING: Unable to save file '$dkey': $err");
                  printf "This was try #$try and it's been %.2f  
seconds since we first tried.  Retrying...\n", tv_interval($start_time);
                  sleep 1;
                  next;
              }
-            $fh->print($chunkbuf{$cn});
-            unless ($fh->close) {
-                error("WARNING: Unable to save file '$dkey'.");
-                printf "This was try #$try and it's been %.2f seconds  
since we first tried.  Retrying...\n", tv_interval($start_time);
-                sleep 1;
-                next;
-            }
              last;
          }
          my $diff = tv_interval($start_time);


-- 
http://develooper.com/ - http://askask.com/