large file investigation..

[Eric Lambrecht]

Brad Fitzpatrick wrote:
> I never designed it for large files, so it's quite likely I do something
> dumb somewhere.  My plan for large files was always to build smarts into
> the client, where a large file is really just dozens/hundreds of
> medium-sized mogile fid.  Never totally got around to it, but you can see
> how mogtool imports huge files like that, so we started.

I think I got it. There were two problems: first was Danga::Socket's 
sysread. If you pass it too big a value for how many bytes you want back 
(200,000,000 for instance), perl would give an 'Out of memory' error and 
quit (!!). I just set a max of 5MB and it doesn't complain.

Second was the Perlbal::ClientHTTP code didn't update the 'alive_time' 
value while a really long PUT was in progress, and so perlbal killed the 
  connection, thinking it had timed out because nothing was being 
written to the client.

Eric...
(very excited to be past these large file issues.. ;-))
-------------- next part --------------
*** ../orig.perl/Danga/Socket.pm        Fri Sep 30 14:50:03 2005
--- Danga/Socket.pm     Mon Oct  3 15:47:34 2005
***************
*** 1019,1025 ****
          }
      }
   
!     my $res = sysread($sock, $buf, $bytes, 0);
      DebugLevel >= 2 && $self->debugmsg("sysread = %d; \$! = %d", $res, $!);
   
      if (! $res && $! != EWOULDBLOCK) {
--- 1019,1025 ----
          }
      }
   
!     my $res = sysread($sock, $buf, ($bytes > 5242880) ? 5242880 : $bytes, 0);
      DebugLevel >= 2 && $self->debugmsg("sysread = %d; \$! = %d", $res, $!);
   
      if (! $res && $! != EWOULDBLOCK) {
-------------- next part --------------
*** ../orig.perl/Perlbal/ClientHTTP.pm  Mon Oct  3 11:50:07 2005
--- Perlbal/ClientHTTP.pm       Mon Oct  3 15:50:26 2005
***************
*** 73,80 ****
--- 73,84 ----
      my Perlbal::ClientHTTP $self = shift;
   
      # see if we have headers?
+
      if ($self->{req_headers}) {
          if ($self->{req_headers}->request_method eq 'PUT') {
+           # possibly very long PUT in progress - don't kill us!
+             $self->{alive_time} = time;
+
              $self->event_read_put;
          } else {
              # since we have headers and we're not doing any special